Audio-visual navigation method combining explicit acoustic geometric reasoning and auxiliary supervision
By combining explicit acoustic geometric reasoning and assisted supervision, the problems of weak geometric perception and insufficient generalization ability of audiovisual navigation in complex environments are solved, thereby improving the navigation ability of intelligent agents in visually limited scenarios and the navigation success rate in unknown environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-24
- Publication Date
- 2026-04-03
AI Technical Summary
Existing audiovisual navigation methods lack geometric interpretation and generalization ability in complex environments, and it is difficult to balance the gradient contribution of tasks during multi-task training, resulting in poor navigation performance.
An explicit acoustic geometric inference module is combined with auxiliary supervision. Geometric representations are extracted through shared encoding and multi-head decoupling structure. An auxiliary supervision mechanism and a dynamic weighting mechanism are introduced to optimize the model training process.
It enhances the model's geometric perception capabilities, improves navigation generalization in unknown environments, and achieves co-evolution of perception and decision-making, thereby improving the robustness of the navigation strategy.
Smart Images
Figure CN121783147A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and robotics, and more specifically, to a cross-modal audiovisual navigation method for sound source localization and path planning in complex environments for embodied agents. Background Technology
[0002] With the development of service robots and embodied intelligence technologies, audio-visual navigation (AV-NaV) has become a research hotspot. AV-NaV aims to guide an intelligent agent to find sound targets in a three-dimensional environment by utilizing information from visual sensors (such as RGB-D cameras) and auditory sensors (such as binaural microphones).
[0003] In existing technologies, mainstream audiovisual navigation methods (such as AV-NaV, SoundSpaces, etc.) mostly employ end-to-end deep reinforcement learning frameworks. While these methods have achieved certain results in specific scenarios, they suffer from the following significant drawbacks when facing complex scenarios with "perceptual limitations" (such as visual blind spots and strong occlusion) and "unknown environments" (such as unfamiliar building layouts):
[0004] 1. Lack of geometric interpretability: Existing models typically treat sound signals as ordinary semantic feature vectors for implicit learning. The model itself is a "black box" and cannot explicitly resolve the distance and direction of the sound source. When visual sensors fail, the model cannot navigate using the physical mechanism of "hearing and positioning" like a biological system.
[0005] 2. Insufficient generalization ability: End-to-end models are prone to overfitting to the visual texture features of the training environment. When the agent enters an unknown test environment or finds an unfamiliar sound source, the navigation success rate will plummet due to the difference in texture feature distribution.
[0006] 3. Multi-task training conflict: When introducing auxiliary tasks for joint training, traditional methods often use fixed weights, which makes it difficult to balance the gradient contributions of the main navigation task and auxiliary tasks. This can easily lead to one task dominating the training, affecting the overall convergence of the model.
[0007] Therefore, there is an urgent need for an audiovisual navigation method that can enhance geometric perception, improve generalization in unknown environments, and adaptively balance multi-task training. Summary of the Invention
[0008] The purpose of this invention is to provide an audiovisual navigation method that combines explicit acoustic geometric reasoning with assisted supervision, so as to solve the problems of weak geometric perception and poor generalization ability in complex environments of existing technologies.
[0009] To achieve the above objectives, the present invention provides the following technical solution: an audiovisual navigation method combining explicit acoustic geometric reasoning and assisted supervision, comprising the following steps: Step S1: Acquire binaural spectrogram data and visual image data of the agent at the current moment; extract auditory feature vectors and visual feature vectors via auditory encoders and visual encoders, respectively. Step S2: Input the auditory feature vectors into the explicit geometric reasoning module (AGR). This module adopts a shared encoding and multi-head decoupling structure, and after processing by a shared feature extraction layer, outputs three sets of features: a high-dimensional geometric representation vector for decision-making, a sound source distance prediction value for assisted supervision, and a sound source azimuth angle prediction value. Step S3: Concatenate and fuse the high-dimensional geometric representation vector and the visual feature vector, inputting it into the recurrent neural network policy unit to generate navigation action commands. Step S4: During the training phase, construct an assisted supervision (AUX) mechanism. Construct an assisted regression loss function using the real sound source coordinates provided by the simulation environment to constrain the explicit geometric prediction of the AGR module. Step S5: Introduce a dynamic weighting mechanism based on uncertainty. The noise levels of navigation and geometry-assisted tasks are automatically evaluated using learnable parameters, the proportions of the weights in the total loss function are dynamically adjusted, and the model parameters are updated jointly.
[0010] The beneficial effects of this invention are as follows:
[0011] 1. Imparting explicit geometric perception to the model: Through the three-way decoupling design of the AGR module, the network is forced to extract physical geometric cues from sound, enabling the agent to still rely on sound even in visually limited scenarios.
[0012] Navigation is achieved through the ability to "identify location by sound".
[0013] 2. Improve generalization ability in unknown environments: The auxiliary supervision mechanism forces the model to learn the essential physical laws of sound wave diffraction and reflection, rather than memorizing environmental textures, thereby significantly improving the model's transfer ability in unknown environments.
[0014] 3. Achieving Co-evolution of Perception and Decision-Making: The dynamic weighting mechanism avoids the blindness of manual parameter tuning, automatically filters geometric noise in complex environments, and ensures the consistency between navigation strategy and geometric perception capabilities.
[0015] Simultaneous improvement. Attached Figure Description
[0016] This invention has a total of appendices Figure 6 Zhang, of which:
[0017] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.
[0018] Figure 2 This is a diagram illustrating the overall model architecture of the method of this invention.
[0019] Figure 3 This is a diagram of the auditory encoder network structure in the method of the present invention.
[0020] Figure 4 This is a diagram of the visual encoder network structure in the method of the present invention.
[0021] Figure 5 This is a diagram of the internal network structure of the AGR in the method of this invention.
[0022] Figure 6 This is a route map for the intelligent agent to find the sound source in the method of the present invention. Detailed Implementation
[0023] Example 1: System Architecture and Data Processing
[0024] This embodiment is built on the SoundSpaces and Habitat simulation platforms.
[0025] 1. Input Data: Auditory Data: The agent receives binaural audio signals from both ears and converts them into binaural spectrograms via Short Time Fourier Transform (STFT). Visual Data: The agent receives RGB images or depth maps.
[0026] 2. Feature Encoding: Auditory Encoder (AudioCNN): Employs a 3-layer Convolutional Neural Network (CNN). The kernel sizes are (8, 8), (4, 4), and (3, 3), with strides of (4, 4), (2, 2), and (1, 1). Visual Encoder (VisualCNN): Also employs a 3-layer CNN structure. The kernel sizes are the same as the auditory encoder, but the strides are (4, 4), (2, 2), and (2, 2). The output features are flattened and mapped to a 512-dimensional feature vector through fully connected layers.
[0027] Example 2: Explicit Geometric Reasoning Module (AGR)
[0028] like Figure 2 As shown, the specific structure definition of the AGR module (AcousticGeometryReasoner) designed in this embodiment is as follows:
[0029] 1. Shared feature extraction layer: Receives auditory feature vector f A It consists of 3 fully connected layers, with the hidden layer dimension set to 256. Each layer is followed by a ReLU activation function and Dropout (dropout rate of 0.1).
[0030] 2. Three-way decoupled output branches: Geometric representation branch: Consists of a linear layer that maps the 256-dimensional shared features to a 128-dimensional high-dimensional geometric representation vector g. tThis vector is directly input into the downstream policy network; the distance regression branch (Distance Head) is used to explicitly predict the sound source distance. The structure is: linear layer (256→128) → ReLU → Dropout → linear layer (128→1), outputting a scalar d. t Azimuth Head Branch: Used for explicit prediction of sound source location. Its structure is the same as the range branch (256→128→1), and the output is a scalar φ. t .
[0031] Example 3: Navigation Decision and Policy Network
[0032] This embodiment uses a GRU (Gated Cyclic Unit) as a state encoder.
[0033] 1. Feature Fusion: The 128-dimensional geometric representation vector g output by AGR is fused. t The 512-dimensional visual features f output by the visual encoder V By concatenating the features along the feature dimension, a 640-dimensional fused feature input is obtained.
[0034] 2. Temporal Decision: The GRU unit receives the fused feature, and its hidden layer dimension is set to 512. The GRU combines the hidden state h from the previous time step. t-1 Output the hidden state h at the current moment. t .
[0035] Example 4: Assisted Supervision and Dynamically Weighted Training
[0036] This embodiment proposes a multi-task loss calculation method based on uncertainty.
[0037] 1. Auxiliary loss function Construction: Normalization: Normalize by dividing the actual distance by the maximum distance of 50.0 meters. Calculation: Calculate the error using Smooth L1 Loss.
[0038] 2. Dynamic Weighting Mechanism: This invention introduces a dynamic weighting algorithm based on homoscedastic uncertainty. Two learnable parameters (log-variance) are defined: s nav and s aux Initialized to 0. Total loss function. The calculation formula is:
[0039]
Claims
1. An audiovisual navigation method combining explicit acoustic geometric reasoning and assisted supervision, characterized in that, Includes the following steps: Step S1: Multimodal perception feature extraction; acquire the binaural spectrogram and visual image of the environment received by the agent at the current time t; use an auditory encoder network to extract features from the binaural spectrogram to obtain the auditory feature vector f. A Visual feature vector f is obtained by using a visual encoder network to extract features from visual images. V ; Step S2: Explicit acoustic geometric reasoning; the auditory feature vector f A The input is fed into a preset explicit geometric reasoning module (AGR); the AGR module adopts a network topology structure of shared coding and multi-head decoupling, mapping the input auditory feature vector into three independent output signals: the first set is a high-dimensional geometric representation vector g. t , used to characterize the potential spatial structure features of a sound source relative to an intelligent agent; The second group contains the predicted distance d from the sound source. t The first group represents the relative Euclidean distance of the sound source for explicit regression; the second group represents the predicted azimuth angle φ of the sound source. t , used for the relative horizontal azimuth of the explicit regressed sound source; Step S3: Cross-modal feature fusion and navigation decision; the high-dimensional geometric representation vector g t With the visual feature vector f V The features are concatenated to form a multimodal fusion feature; this fusion feature is then input into the recurrent neural network policy unit to generate the navigation action command 'a' for the current moment. t And drive the intelligent agent to execute; Step S4: Construct auxiliary supervised training objectives; during the model training phase, obtain the true distance label d of the sound source from the simulation environment. true and the actual azimuth label φ true Construct an auxiliary regression loss function to calculate the predicted distance d. t Distance label d true The distance loss between them, and the predicted azimuth angle φ t With the actual azimuth label φ true Azimuth angle loss between; Step S5: Dynamic weighted optimization based on uncertainty for multiple tasks; construct a total loss function that includes navigation strategy loss and the auxiliary regression loss; introduce learnable uncertainty parameters, dynamically adjust the weights of each task loss during training, and jointly update the network parameters through the backpropagation algorithm.
2. The method according to claim 1, characterized in that, The specific network structure of the explicit geometric reasoning module mentioned in step S2 is as follows: Shared feature extraction layer: Composed of a multi-layer fully connected network (MLP), each layer contains linear transformation, activation function (ReLU) and random deactivation (Dropout) operations, used to extract implicit physical geometric cues from raw auditory features; Geometric Representation Branch: Connected after the shared feature extraction layer, it contains a linear mapping layer for outputting the high-dimensional geometric representation vector g. t This vector directly participates in downstream navigation decisions; Distance Regression Branch: Independent of the geometric representation branch, it consists of a linear layer, an activation function layer, a random deactivation layer, and an output linear layer connected in sequence, used to regress the source distance prediction value in scalar form; Azimuth regression branch: Independent of the geometric representation branch, with the same structure as the distance regression branch, it is used to regress the scalar form of the sound source azimuth prediction value.
3. The method according to claim 1, characterized in that, The process of constructing the auxiliary regression loss function in step S4 includes: Target normalization: normalize the true distance label d true Normalization is performed by dividing by the preset maximum detection range; the true azimuth label φ is then calculated using trigonometric functions. true and azimuth prediction value φ t Mapping to a continuous space to eliminate the effects of periodic discontinuities in angles; Loss Calculation: The normalized distance error and azimuth error are calculated using the Smooth L1 Loss function, and the weighted sum of the two is used to obtain the auxiliary regression loss.
4. The method according to claim 1, characterized in that, The specific method for multi-task dynamic weighted optimization based on uncertainty described in step S5 is as follows: Pre-set a set of learnable log-variance parameters s = {s1, s2, ..., s...} K }, each corresponding to one of K different training tasks, where s i Characterizes the arbitrary uncertainty of the i-th task; Define the weighted total loss function for: in, This represents the original loss value for the i-th task; During training, the total loss function is minimized. The network automatically increases the parameter s corresponding to high-noise tasks. i This reduces the loss items of the task. The weight exp(-s i This enables the coordinated convergence of perception and decision-making tasks.
5. The method according to claim 1, characterized in that, In step S1: The auditory encoder network is a convolutional neural network containing multiple two-dimensional convolutional layers (Conv2d) for processing binaural spectrogram data containing binaural time difference and intensity difference; The visual encoder network is a convolutional neural network containing multiple convolutional layers and fully connected layers, used to extract semantic features from RGB images or depth images.
6. The method according to claim 1, characterized in that, In step S3: The recurrent neural network strategy unit adopts a gated recurrent unit (GRU); this unit receives the multimodal fusion features at the current time and the hidden state at the previous time, and outputs the hidden state at the current time to memorize historical path information; The navigation action commands are in a discrete action space, including forward, left turn, right turn, and stop.
7. An audiovisual navigation system for implementing the method according to any one of claims 1 to 6, characterized in that, include: Perception module: configured to collect environmental data and preprocess it to generate binaural spectrograms and visual feature maps; Explicit geometric inference unit: It has a built-in shared encoder and multi-head decoupling network, configured to receive auditory features and output geometric representation vectors for decision-making and predicted physical parameters for supervision in parallel; Decision control unit: configured to fuse multimodal features and generate navigation control commands based on a recurrent neural network; Auxiliary supervised training unit: configured to calculate auxiliary loss using privileged geometric information provided by the simulation environment, and dynamically balance gradient updates for navigation and perception tasks using an uncertainty weighting mechanism.