Dynamic audio-visual navigation method based on intensity calibration geometric coding and cross-modal scene aggregation

Abstract

The invention provides a dynamic audio-visual navigation method based on intensity calibration geometric coding and cross-modal scene aggregation. The dynamic audio-visual navigation method is suitable for application scenes such as service robots, security inspection and virtual reality interaction. According to the method, a depth visual image and a binaural spectrogram serve as input, firstly, a sound intensity map and depth geometric information are explicitly fused through an intensity calibration geometric encoder, and physical-space correlation across sound categories is modeled; then, a cross-modal scene aggregator is utilized to convert the multiple groups of spatial features into a Token sequence, fine-grained and content-aware multi-modal dynamic fusion is realized through Transform, and global state representation rich in reasoning ability is generated; and finally, training a navigation strategy of time sequence perception under a near-end strategy optimization framework based on the representation, and outputting a tracking action sequence aiming at the mobile sound source. According to the method, the navigation success rate and the path efficiency are remarkably improved under the conditions of a mobile sound source and unknown sound, and the generalization ability and the decision robustness of the system in a dynamic environment are enhanced.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, robot navigation and multimodal perception technology. Specifically, it relates to a method for achieving robust audiovisual navigation in a dynamic sound source environment, and more particularly to a dynamic audiovisual navigation system that models the sound intensity-geometric physical correlation through an intensity calibration geometric encoder and uses a cross-modal scene aggregator to achieve fine-grained multimodal fusion. Background Technology

[0002] With the development of embodied intelligence and autonomous robotics, intelligent agents capable of completing goal-oriented tasks in complex environments have become a research hotspot. Audio-visual navigation, by fusing visual and auditory information, enables intelligent agents to locate and navigate to sound sources in visually limited or occluded scenarios, significantly expanding the perceptual capabilities of traditional visual navigation. Existing research mainly focuses on static sound source scenarios and has made some progress. However, sound sources in real-world environments are often dynamically moving, such as walking people, moving vehicles, or equipment, which constitutes a more challenging mobile audio-visual navigation task. In this task, the sound source location continuously changes, and the intelligent agent needs to infer the target location and make effective decisions in real time from dynamically changing multimodal cues. Current methods face two major limitations when dealing with moving sound sources: first, over-reliance on sound semantic features learned from training data leads to a sharp decline in the model's generalization performance when facing unfamiliar sound categories; second, multimodal fusion strategies are mostly simple feature splicing or late fusion, making it difficult to capture the fine-grained spatial correspondences between audio-visual cues, limiting the ability to accurately reason about and plan states in dynamic environments. Therefore, there is an urgent need for a dynamic audiovisual navigation method that can model cross-category physical laws and realize fine-grained multimodal interaction fusion, so as to improve the navigation robustness and generalization performance of intelligent agents under moving sound sources and unknown sound conditions. Summary of the Invention

[0003] This invention provides a dynamic audiovisual navigation method based on intensity-calibrated geometry encoding and cross-modal scene aggregation, termed the ICGE-CMSA navigation framework. This method aims to improve the generalization ability, state reasoning accuracy, and path planning robustness of audiovisual navigation in environments with moving sound sources, making it particularly suitable for service robots, security patrols, and virtual reality interaction. Compared to existing methods, this invention introduces the following core innovations: 1. Intensity-calibrated geometry encoder (ICGE): Unlike traditional encoding methods that rely on semantic features of sound, the ICGE module explicitly models the physical relationship between sound intensity distribution and scene geometry by spatially aligning and channel-by-channel stitching sound intensity maps with depth visual images. This encoder learns to recognize cross-sound category-invariant "high sound intensity-geometric structure" co-occurrence patterns, providing semantically independent generalization priors for navigation strategies and significantly improving adaptability to unfamiliar sounds. 2. Cross-Modal Scene Aggregator (CMSA): Based on the Transformer architecture, CMSA unfolds multiple sets of spatially structured feature maps into token sequences, introducing learnable fusion tokens and achieving fine-grained, content-aware cross-modal interaction through a multi-head self-attention mechanism. This module can dynamically integrate visual, auditory, geometric, and intensity calibration information to generate a global state representation rich in spatial reasoning capabilities, overcoming the problems of information loss and rigid fusion in traditional vector stitching methods. 3. Multi-level Feature Extraction and Map Construction Strategy: The system constructs a geometric occupancy map and a sound intensity map in parallel, encoding the scene traffic structure and the spatiotemporal distribution of sound intensity, respectively. A multi-scale convolutional encoder extracts basic visual, audio, spatial alignment, and intensity calibration features, forming a multimodal input that combines local details and physical laws, providing a rich and structured information foundation for subsequent deep fusion. 4. Temporally Aware Policy Learning and Optimization Mechanism: The fused state representation is used for temporal modeling via a GRU network, then input into an Actor-Critic network for training within a proximal policy optimization framework. The reward function integrates success rewards, progressive distance rewards, and time penalties to guide the agent in efficiently tracking moving sound sources in dynamic environments. The system employs end-to-end joint optimization to ensure the co-evolution of state representation and policy learning, further enhancing the stability and generalization performance of navigation decisions. Attached Figure Description

[0004] Figure 1 The overall system architecture diagram of the ICGE-CMSA navigation framework includes processes such as observation input, multi-scale feature extraction, cross-modal scene aggregation, and policy learning. Detailed Implementation

[0005] The implementation of this invention includes the following four main stages: multimodal feature extraction and map construction, intensity calibration geometric encoding, cross-modal scene aggregation and state fusion, and temporal policy learning and system optimization. 1. Multimodal Feature Extraction and Map Construction Stage: At each time step, the agent receives a depth visual image and a binaural spectrogram as input. The depth image extracts 512-dimensional visual features through a multi-layer convolutional encoder. The binaural spectrogram is obtained through a short-time Fourier transform and used to calculate sound intensity values ​​in real time. Simultaneously, the system constructs two types of auxiliary maps in parallel: a geometric occupancy map centered on the agent is generated based on depth information, encoding the passable structures and obstacle distribution of the scene; a sound intensity map is generated based on accumulated historical sound intensity observations, recording the spatiotemporal distribution of sound source energy in the explored area. These maps provide structured spatial representations for subsequent cross-modal alignment and physical law modeling. 2. Intensity Calibration Geometric Encoding Stage: The sound intensity map is upsampled to the same spatial resolution as the depth image through bilinear interpolation, achieving pixel-level spatial alignment. Subsequently, the upsampled sound intensity map and the original depth map are concatenated along the channel dimension to form a fused input tensor. This tensor is input to a dedicated depth convolutional encoder, which extracts intensity calibration geometric features through end-to-end training. This feature explicitly models the physical relationship between sound intensity distribution and scene geometry (such as walls, doorways, and open areas), learning a cross-sound category-invariant "high intensity-geometric structure" co-occurrence pattern, providing generalizable physical prior knowledge for the navigation system. 3. Cross-modal scene aggregation and state fusion stage: The basic visual features, basic audio features, spatial alignment features, and intensity calibration geometric features retain their spatial structure (size 5×5×256) and are each expanded into 25 256-dimensional token sequences. A learnable global fusion token is introduced, which, together with all modal tokens, constitutes the input sequence, and a learnable positional encoding is added. This sequence is input to an N-layer Transformer encoder, which achieves fine-grained interaction and information integration across modalities and spaces through a multi-head self-attention mechanism. Finally, the output vector of the corresponding fused token is taken as the global state representation of the current time step. This representation integrates multimodal observations, spatial relationships, and physical laws, possessing powerful discrimination and reasoning capabilities. 4. Temporal Policy Learning and System Optimization Stage: The global state representation is input into a one-way gated recurrent unit (GRU) for temporal modeling, capturing the dynamic evolution of the moving sound source's state and outputting a decision state vector fused with historical information. This state vector is input into the Actor network and the Critic network respectively: the Actor network outputs the probability distribution of four discrete actions: forward, left turn, right turn, and stop; the Critic network evaluates the expected cumulative reward of the current state. A proximal policy optimization algorithm is used for training, and the reward function includes a successful arrival reward, an asymptotic reward for reducing the distance to the target, and a time penalty to encourage efficient navigation.The entire system undergoes end-to-end joint training in a simulation environment containing both static and dynamic sound sources, and is tested and validated on unseen scenes and sound categories. Furthermore, the system supports parametric configuration, allowing users to adjust hyperparameters such as the depth of ICGE and CMSA, the number of Transformer layers, and the weights of the reward function according to task complexity, adapting to navigation scenarios with varying dynamic characteristics and real-time requirements. This invention can be widely applied in fields requiring robust dynamic target tracking, such as home service robots, security monitoring systems, and virtual reality interactive platforms.

Claims

1. A dynamic audiovisual navigation method based on intensity calibrated geometric coding and cross-modal scene aggregation, comprising: receiving a depth visual image and a binaural spectrogram input from the environment; constructing a geometric occupancy map and a sound intensity map as auxiliary spatial representations; spatially aligning the sound intensity map and the depth visual image through an intensity calibrated geometric encoder, and fusing features from both modalities to extract intensity calibrated geometric features; transforming the base visual features, the base audio features, the spatially aligned features, and the intensity calibrated geometric features into Token sequences through a cross-modal scene aggregator, and performing cross-modal interaction and fusion using a Transformer encoder to generate a global state representation; inputting the state representation into a temporal modeling network to perform policy learning, and outputting a sequence of navigation actions for the moving sound source.

2. The dynamic audio-visual navigation method based on intensity calibrated geometric coding and cross-modal scene aggregation of claim 1, wherein the intensity calibrated geometric encoder performs the following steps: upsampling the sound intensity map to the same resolution as the depth visual image; concatenating the upsampled sound intensity map and the depth map in the channel dimension; and extracting features through a convolutional encoder to model the cross-category physical correlation between the sound intensity distribution and the scene geometry.

3. The dynamic audio-visual navigation method based on intensity calibrated geometric coding and cross-modal scene aggregation of claim 1, wherein the cross-modal scene aggregator performs the following steps: expanding each modality feature map into a Token sequence and adding learnable position encodings; introducing learnable global fusion Tokens; implementing cross-modal multi-head self-attention interaction through a multi-layer Transformer encoder; and taking the output of the fusion Tokens as the global state representation.

4. The dynamic audio-visual navigation method based on intensity calibrated geometric coding and cross-modal scene aggregation of claim 1, wherein the temporal modeling network is a gated recurrent unit used to sequence model the global state representation and output a decision state vector that integrates historical information.

5. The dynamic audio-visual navigation method based on intensity calibrated geometric coding and cross-modal scene aggregation of claim 1, wherein the policy learning adopts a proximal policy optimization framework based on an Actor-Critic network structure, and the reward function includes a success reward, a distance reduction reward, and a time penalty.

6. The dynamic audio-visual navigation method based on intensity calibrated geometric coding and cross-modal scene aggregation of claim 1, wherein the geometric occupancy map is constructed based on the depth visual image and encodes the distribution of passable areas and obstacles in the scene; and the sound intensity map is generated based on historical sound intensity observations and records the spatiotemporal distribution of sound source energy in the explored area.

7. The dynamic audiovisual navigation method based on intensity calibrated geometric coding and cross-modal scene aggregation as claimed in claim 1 wherein further comprising: In the training phase, end-to-end joint optimization is adopted, multiple rounds of training are performed in a simulated environment containing static and dynamic sound sources, and generalization performance is evaluated on unseen scenes and sound categories.

Images