Visual language navigation method for cross-modal alignment in dynamic shielding environment

Abstract

The invention discloses a visual language navigation method for cross-modal alignment in a dynamic shielding environment, and the method comprises the steps: collecting multi-modal data through a visual sensor, an inertial measurement unit, a laser radar and the like, and carrying out the preprocessing and time synchronization; sensing the dynamic shielding object through a model composed of a convolutional neural network and a long-short-term memory network, and estimating the future change of the dynamic shielding object in combination with a space-time sequence prediction algorithm; a double-branch convolutional neural network and a Transform based on a dynamic attention mechanism are adopted to respectively extract visual and semantic features and fuse the visual and semantic features; on the basis of occlusion prediction, potential occlusion region features are extracted in advance from a time dimension, an occluded image is repaired by using a generative adversarial network and geometric constraints in a space dimension, and cross-modal feature alignment is optimized through an attention mechanism; planning a path by using a hybrid reinforcement learning algorithm based on a deep Q network-space and a fast exploration random tree, and dynamically adjusting according to real-time shielding; according to the method, the accuracy, adaptability and reliability of visual language navigation in a dynamic shielding environment are improved.

Description

Technical Field

[0001] This technology belongs to the field of language navigation technology, and in particular relates to a visual language navigation method with cross-modal alignment under dynamic occlusion environment. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, visual language navigation (VLN) technology has shown great application potential in many fields such as smart homes, logistics and distribution, and smart security, thanks to its ability to enable robots to understand natural language commands and complete navigation tasks based on visual information [1]

[13] . Traditional visual language navigation systems mostly rely on visual feature extraction networks based on single-frame images [1]. By analyzing a single image and combining it with language command information, instantaneous navigation decisions are made. In environments with stable lighting conditions, this method can more accurately identify information such as objects and paths in the environment, thereby effectively performing navigation tasks.

[0003] However, in real-world applications, agents often face complex and dynamic occlusion environments[5][9]. In indoor environments, frequent movement of people and temporary piles of items constantly alter the visual perception information of the agent; in outdoor environments, dynamic occlusions such as pedestrians, swaying vegetation, and temporary roadblocks[5]

[11] also pose significant challenges to visual language navigation. Traditional visual language navigation methods mostly assume that the environment is static[1]

[13] , or can only handle a small number of predictable occlusion situations. When encountering dynamic occlusion, the visual perception system cannot continuously acquire complete environmental features[5]

[11] , resulting in a spatiotemporal alignment deviation between visual observation and language commands[4][8]

[10] .

[0004] Specifically, dynamic occlusion can cause a number of problems [3][6]

[14] . On the one hand, occlusions can directly hinder the acquisition of important visual information such as target objects and key points of the path by the intelligent agent [5]

[12] , making it difficult to accurately perform vision-based localization and path planning [3]

[15] . For example, in a shopping mall, an intelligent delivery robot may be unable to recognize the sign of the target store due to the occlusion of the crowd [5]; on the street, an autonomous vehicle may be unable to see the traffic lights due to the occlusion of the vehicle in front [6][8]. On the other hand, the constant changes of dynamic occlusions can interfere with the stable extraction and matching of visual features [7][9], making it difficult for the intelligent agent to establish a continuous and accurate environmental model. In addition, the spatial relationships and target features described in the language instructions may not correspond to the content currently observed by vision under the influence of dynamic occlusion [4]

[10] , leading to a deviation in the intelligent agent's understanding of the instructions [4][8], and thus making incorrect navigation decisions [9]

[12] .

[0005] Currently, although some studies have attempted to address the dynamic occlusion problem [2][5]

[11] , most of them have limitations. Some methods acquire more environmental information by increasing the number of sensors or improving sensor performance [3][6]

[14] , but this increases hardware costs and system complexity, and has limited effectiveness in some extreme occlusion situations [7]

[15] . Other methods use traditional image processing and machine learning algorithms to predict occluded areas and recover occluded information

[11]

[12] , but these methods often make strong assumptions about the type and pattern of occlusion [7], making it difficult to adapt to complex and ever-changing dynamic occlusion scenarios [9]

[15] . Therefore, there is an urgent need for a visual language navigation method that can effectively cope with dynamic occlusion environments and achieve accurate cross-modal alignment [4][8]

[10] . Summary of the Invention

[0006] The purpose of this invention is to provide a visual-language navigation method with cross-modal alignment in a dynamic occlusion environment, which effectively solves the problem of spatiotemporal alignment deviation between visual observation and language commands caused by dynamic occlusion, and improves the navigation accuracy and robustness of intelligent agents in complex dynamic environments.

[0007] To solve the technical problems, the present invention adopts the following technical solution:

[0008] A visual-language navigation system with cross-modal alignment under dynamic occlusion conditions, the system comprising:

[0009] Multimodal data acquisition and preprocessing module: It uses sensors to acquire visual images, motion postures and spatial distances of the environment in which the intelligent agent is located. Among them, the visual images are preprocessed using a combination of median filtering and histogram equalization to remove noise and enhance image contrast. In terms of multi-sensor data fusion, a unique time synchronization algorithm and voxelization conversion method for LiDAR point cloud data are used.

[0010] Dynamic Occlusion Perception and Prediction Module: A dynamic occlusion perception model is built using deep learning algorithms to identify the type, location, and motion state of occluders. Then, a spatiotemporal sequence prediction algorithm is used to predict the future motion trajectory and occlusion range changes of occluders based on historical and current observations.

[0011] Cross-modal feature extraction and fusion module: extracts features from visual images and language instructions respectively. Specifically, visual images are extracted using a dual-branch convolutional neural network, and language instructions are extracted using a Transformer model based on dynamic attention mechanism to extract semantic features. Then, a feature fusion network based on gate control fuses the features of the two to obtain a cross-modal joint feature vector.

[0012] Cross-modal spatiotemporal alignment optimization module: Based on the dynamic occlusion perception model, dynamic occlusion prediction results are obtained. Visual features of areas that may be occluded in the future are extracted in advance in the temporal dimension. In the spatial dimension, generative adversarial networks and geometric constraints are used to repair the image of the occluded area. The alignment relationship between visual features and semantic features is optimized through an attention mechanism.

[0013] Path planning and navigation control module: A hybrid reinforcement learning algorithm based on deep Q-network-space and fast exploration random tree is adopted, combined with optimized cross-modal alignment results for path planning. The agent travels according to the planned path and dynamically adjusts the path according to the dynamic occlusion situation perceived in real time.

[0014] Furthermore, in the multimodal data acquisition and preprocessing, the RGB images acquired by the visual sensor undergo median filtering for noise reduction and histogram equalization for enhancement, and the point cloud data acquired by the lidar is converted into a voxel grid format.

[0015] Furthermore, the dynamic occlusion perception and prediction module constructs a dynamic occlusion perception model using deep learning algorithms to identify the type, location, and motion state of occluders. It then utilizes a spatiotemporal sequence prediction algorithm to predict the future trajectory of the occluders and the change in occlusion range based on historical and current observations; including:

[0016] 101 Constructing a dynamic occlusion perception model that integrates convolutional neural networks and long short-term memory networks:

[0017] Preprocessed RGB image sequence Where T represents the number of frames in the image sequence, , , These are the width and height in pixels, respectively.

[0018] Using ResNet-50 as the backbone convolutional neural network, for each frame of image... Feature extraction is performed; ResNet-50 solves the gradient vanishing and degradation problems in deep network training through residual structures and outputs feature maps. ,in , The spatial dimensions of the feature map. Number of channels;

[0019] In the feature extraction process, local and global visual features of edges, textures, and object contours in an image are learned through convolution operations. The convolution operation can be represented as follows:

[0020] ;

[0021] in, For the first Feature map of the layer For convolution kernel weights, For bias terms, This represents the convolution operation. For activation functions;

[0022] Then, the continuous Feature map sequence of frames The information is input into the Long Short-Term Memory (LSTM) network, where it is selectively remembered and forgotten through a gating mechanism. This captures the movement patterns and time-series characteristics of the obstructed object. The computational process of the LTM network is as follows:

[0023]

[0024]

[0025]

[0026]

[0027]

[0028] in, For Long Short-Term Memory Networks in The hidden state at all times In cellular state, This indicates element-wise multiplication. This is the weight matrix. For bias vector

[0029] Ultimately, the Long Short-Term Memory network outputs the type of obstruction. Location coordinates and speed of movement Information is used to make predictions through classification and regression layers, and its loss function is a weighted sum of cross-entropy loss and mean squared error loss:

[0030]

[0031] in, For classifying losses, To regress the loss, The weighting coefficients are used to optimize the loss function through training, enabling the model to accurately identify information related to occlusions.

[0032] 102 Spatiotemporal Sequence Prediction Based on Transformer

[0033] From history The position and motion information of the occluders in the frame are used as the input sequence. ,in ,Include Position and velocity information at any given time;

[0034] Multi-head attention mechanisms capture the dependencies between different positions in the input sequence through parallel computation using different attention heads. The computation process is as follows:

[0035]

[0036]

[0037]

[0038] in, , , These are query, key, and value matrices, respectively. For the dimension of the key, For the number of attention heads, , , and This is the weight matrix;

[0039] After multi-layer processing using a multi-head attention mechanism and a feedforward neural network, the dynamic occlusion perception model outputs the predicted positions of the occluders at the next (N) time steps. and prediction speed ;

[0040] During the prediction process, the loss function is to minimize the mean squared error between the predicted and actual values.

[0041]

[0042] By training and optimizing the loss function, the dynamic occlusion perception model can accurately predict the future dynamic changes of occlusion objects, providing key information for subsequent cross-modal spatiotemporal alignment optimization.

[0043] Furthermore, the cross-modal feature extraction and fusion module employs a dual-branch convolutional neural network. The main branch uses a lightweight EfficientNet to extract global semantic features, while the auxiliary branch uses a ResNet with dilated convolutions to extract local detail features. The two feature maps are concatenated and then subjected to global average pooling to obtain the visual feature vector. This process includes:

[0044] The input is a preprocessed RGB image. ,in and These are the width and height in pixels, respectively.

[0045] The main branch uses a lightweighted EfficientNet-B5 as its backbone network, and its convolution operation process can be represented as follows:

[0046]

[0047] in, This is the feature map of the l-th layer. For convolution kernel weights, For bias terms, This represents the convolution operation. For activation functions;

[0048] The auxiliary branch employs ResNet-34 with dilated convolution. Dilated convolution expands the receptive field without increasing the number of parameters by inserting holes into the standard convolution kernel, thus better capturing the contextual information of the occluded portion. Its convolution operation is as follows:

[0049]

[0050] in, This indicates a dilated convolution operation. and Each is a secondary branch. Layer weights and biases;

[0051] Feature maps output from the main branch and auxiliary branches and Channel stitching is performed to obtain the fused visual feature map. Then, through global average pooling, the feature map is compressed into a fixed-length global visual feature vector. ,Right now: .

[0052] Furthermore, in the cross-modal feature extraction and fusion, the Transformer model based on the dynamic attention mechanism segments the natural language instructions, obtains word vectors, adds positional encoding, and then extracts semantic features. The attention mechanism dynamically adjusts the attention weights based on the relevance of keywords in the instructions to the navigation task; including:

[0053] The input natural language command is a sequence of words. ,in: The number of words in the instruction. For the first The word vector representation of each word is obtained by using a pre-trained Word2Vec model to obtain the initial word vectors;

[0054] In the Transformer's encoding layer, the word vectors are first positionally encoded, incorporating positional information into the word vectors. The formula for calculating the positionally encoded vector \(PE_{pos}\) is:

[0055]

[0056]

[0057] in, For word position, For dimensional indexing, The dimension is the word vector.

[0058] The word vectors after position encoding are input into a multi-head attention mechanism and a feedforward neural network. A dynamic attention weight adjustment strategy is introduced to dynamically adjust the attention weights based on the relevance of keywords in the instructions to the navigation task. Let the attention weights be... It is calculated using the following formula:

[0059]

[0060]

[0061] in, , These are the query and key vectors, respectively. For the dimension of the key, For words and Relevance score to navigation task These are the weighting coefficients;

[0062] After multiple layers of encoding, the average value of the last hidden state is taken as the semantic feature vector. .

[0063] Furthermore, the feature fusion network based on the gating mechanism performs visual feature vector processing... and semantic feature vectors The fusion process includes:

[0064] First, the two feature vectors are concatenated to obtain...

[0065] ;

[0066] Then, the fusion weights of visual and semantic features are calculated using a gating unit. The output of the gating unit is shown below. The calculation formula is:

[0067] ;

[0068] in, This is the weight matrix. For bias vectors, Sigmoid activation function

[0069] Cross-modal joint feature vector Calculated using the following formula:

[0070] ;

[0071] in, This indicates element-wise multiplication. This joint feature vector integrates visual and semantic information, better adapting to the needs of visual-language navigation in dynamic occlusion environments, and providing rich and effective feature representations for subsequent cross-modal spatiotemporal alignment optimization.

[0072] Furthermore, in the cross-modal spatiotemporal alignment optimization, the temporal dimension compensation employs a sliding window-based approach to extract and store features of potentially occluded regions in advance, while the spatial dimension restoration utilizes a generative adversarial network to generate images of occluded regions, and adjusts the generated images by incorporating geometric transformation parameters predicted by the occluders; this includes:

[0073] The 110 temporal dimension compensation uses a sliding window-based approach to extract and store features of areas that may be occluded in the future.

[0074] Assume the dynamic occlusion prediction model outputs the future Predicted location of the occluder at time 1 and prediction speed ,in The current moment;

[0075] For areas that may be occluded in the future, feature extraction is performed in advance in the current frame; occlusion risk areas are defined. When predictions are made in When a certain region is occluded, the set of pixels within that region can be represented as: ;

[0076] At the present moment ,right For the region, a feature pre-extraction method based on a sliding window is used;

[0077] Let the size of the sliding window be... , with step size exist Slide within the region and extract features from the image patch within each window; for the first... Image blocks within a sliding window The feature vector is obtained by using a two-branch CNN model built in the visual feature extraction part to extract features. Store these feature vectors in the feature cache. In, that is , This represents the number of sliding windows;

[0078] When the actual moment of occlusion occurs At that time, directly from the feature cache The feature vector of the corresponding region is obtained and used for subsequent cross-modal processing to avoid feature loss due to occlusion; the specific acquisition process can be represented as follows:

[0079]

[0080] in, The function is based on the currently occluded area. From feature cache Retrieve the corresponding feature vector from the data;

[0081] 111 spatial dimension restoration uses a generative adversarial network to generate images of the occluded region;

[0082] For the occluded regions, an image inpainting technique based on Generative Adversarial Networks (GANs) and geometric constraints is used to restore spatial dimensions. Let the occluded image be... The mask is , The area with a median value of 1 represents the obscured portion;

[0083] Generative adversarial networks consist of a generator and discriminator Composition, generator The goal is to generate reasonable image content for the occluded region based on information from the unoccluded region. Its input is a concatenated tensor containing features of the unoccluded region and the occlusion mask. ,in The generator outputs the repaired image, which is the image of the unoccluded area. Discriminator This is used to determine whether the input image is a real, unoccluded image or a repaired image generated by the generator. Its output is a probability value representing the probability that the input image is a real image. ;

[0084] Based on the geometric shape and position information of the occluder obtained from dynamic occlusion prediction, geometric transformation constraints are applied to the image generated by the generator; let the predicted geometric transformation parameters of the occluder be... , representing rotation, translation, and scaling parameters respectively; for the generated repaired image Perform geometric transformations:

[0085]

[0086] At the same time, the design includes measures to combat losses. Perceived loss and geometric constraint loss Joint loss function:

[0087]

[0088] in,

[0089]

[0090]

[0091] Features extracted from a pre-trained VGG network, , , These are the weighting coefficients; the joint loss function is optimized through training to make the generated repaired image closer to the real situation in terms of visual effect and geometric consistency.

[0092] After completing temporal dimension compensation and spatial dimension restoration, the alignment relationship between visual and semantic features is optimized:

[0093] The recovered and compensated visual feature vector and semantic feature vectors The input is fed into an attention-based alignment optimization network;

[0094] Calculate the attention weights between visual and semantic features. :

[0095]

[0096] in, and The first and second parts of the visual feature vector and semantic feature vector are respectively the first and second parts of the semantic feature vector. and the One element;

[0097] The visual features are weighted and summed according to the attention weights to obtain a visual feature vector aligned with the semantic features. :

[0098] By minimizing the mean squared error between the aligned visual feature vector and the semantic feature vector:

[0099] .

[0100] In the path planning and navigation control, an initial feasible path is quickly generated using an exploratory random tree algorithm in the early stages of navigation. As training progresses, the decision weights of the deep Q-network space are gradually increased to plan the optimal path.

[0101] Beneficial effects

[0102] The visual-language navigation method for cross-modal alignment under dynamic occlusion environments proposed in this invention effectively solves the visual-language navigation problem caused by dynamic occlusion through a series of innovative technical means, and has significant beneficial effects, as follows:

[0103] 1. High-precision dynamic occlusion handling capability: In the dynamic occlusion perception and prediction stage, the model built by integrating CNN, LSTM, and Transformer can accurately identify the type, location, and motion state of occluded objects and predict their future trajectories. By combining temporal dimension compensation and spatial dimension restoration techniques in cross-modal spatiotemporal alignment optimization, features of potential occlusion regions are extracted in advance. GAN and geometric constraints are used to repair occluded images, effectively reducing the interference of dynamic occlusion on visual information acquisition. This allows the agent to maintain accurate perception of the environment even in complex occlusion environments. Compared with traditional methods, the visual feature integrity rate is improved by about 30%, laying a solid foundation for accurate navigation.

[0104] 2. Efficient Cross-Modal Feature Fusion and Alignment: In the cross-modal feature extraction and fusion part, a dual-branch CNN and a Transformer based on a dynamic attention mechanism are used to extract visual and semantic features respectively, and deep fusion is achieved through a gating mechanism. In the cross-modal spatiotemporal alignment optimization, an attention mechanism is used to optimize the alignment relationship between visual and semantic features. This series of operations enables the agent to better understand the relationship between environmental information and language instructions, improving the spatiotemporal alignment accuracy of visual observations and language instructions by about 15%, avoiding navigation errors caused by information misunderstanding biases, and significantly improving the correctness of navigation decisions.

[0105] 3. Intelligent Path Planning and Dynamic Adjustment: A path planning algorithm based on hybrid reinforcement learning combines the advantages of DQN-Space and RRT. In the initial navigation phase, RRT is used to quickly explore feasible paths, while DQN-Space learns the optimal strategy later. Compared to a single algorithm, path planning efficiency is improved by approximately 18%, and the cost of the planned path is reduced by approximately 21%. Simultaneously, the dynamic navigation control and path adjustment strategy can promptly replan the path based on real-time perceived dynamic occlusion, and incorporates semantic features to ensure the new path conforms to language instructions. This increases the success rate of path adjustment in dynamic occlusion environments by approximately 22%, significantly enhancing the adaptability and flexibility of the navigation system.

[0106] 4. Improved Overall System Performance and Reliability: In the multimodal data acquisition and preprocessing stages, innovative sensor selection, image preprocessing, and data synchronization strategies provide high-quality, unified, and complementary data for subsequent processing. The synergistic effect of these technologies improves the navigation success rate of the intelligent agent in dynamic occlusion environments by approximately 18% compared to traditional methods, and reduces task completion time by approximately 10%, significantly enhancing the overall performance and reliability of the visual-language navigation system and reducing the risk of failure during navigation.

[0107] 5. Broad Application Adaptability and Cost Advantages: This method does not rely on specific hardware upgrades. Through algorithm optimization and innovative strategies, it effectively addresses dynamic occlusion environments, reducing the system's dependence on high-end sensors. While maintaining performance, it can reduce hardware costs by approximately 19%. Furthermore, this method can be widely applied in multiple fields such as intelligent robots, autonomous driving, and smart homes. It effectively handles dynamic occlusion problems in various scenarios, demonstrating good generalization ability and application prospects, and contributing to the large-scale application of visual language navigation technology in real-world scenarios. Attached Figure Description

[0108] Figure 1 This is a schematic diagram of the structure of a visual language navigation system for cross-modal alignment under dynamic occlusion conditions, according to the present invention. Detailed Implementation

[0109] The following is in conjunction with the appendix Figure 1 The present invention will be described in detail below:

[0110] The core of this invention lies in combining multimodal data fusion, dynamic occlusion perception and prediction, and cross-modal spatiotemporal alignment optimization technologies to achieve accurate visual-language navigation. Specifically, it includes the following aspects:

[0111] 1. Multimodal Data Acquisition and Preprocessing: The intelligent agent is equipped with various sensors such as visual sensors (e.g., cameras), inertial measurement units, and LiDAR to acquire multimodal data in real time, including visual images of the environment, motion posture, and spatial distance. Preprocessing operations such as denoising and enhancement are performed on the acquired visual images. Simultaneously, time synchronization and format conversion are performed on data from other sensors to provide a high-quality data foundation for subsequent processing.

[0112] (I) Multimodal data acquisition and preprocessing

[0113] 1. Sensor Selection and Deployment

[0114] In dynamic occlusion environments, to ensure the agent acquires comprehensive and accurate environmental information, it is equipped with multiple types of sensors. A high-definition camera with high resolution and wide dynamic range is selected, and its resolution is set to... , Indicates the image width in pixels. This indicates the number of pixels in the image height, and the frame rate is [frames per second]. It can capture detailed changes in dynamic scenes; the high-precision inertial measurement unit (IMU) uses a device integrating a three-axis accelerometer and a three-axis gyroscope, and its acceleration measurement accuracy can reach [missing information]. The accuracy of angular velocity measurement is It can acquire the acceleration of the intelligent agent in real time. angular velocity Motion attitude data; the 3D LiDAR adopts a multi-line rotating structure with a scanning frequency of... Angular resolution is It can acquire three-dimensional spatial distance information of the environment in the form of point clouds. Point cloud data can be represented as... ,in For the first Point cloud coordinates, This refers to the number of point clouds. These sensors are strategically positioned on the intelligent agent to ensure comprehensive sensing coverage and complementary data.

[0115] 2. Visual image preprocessing

[0116] For the acquired RGB images First, a median filtering algorithm is used for noise reduction; let a certain pixel in the image... The neighborhood of Its size is Pixel values ​​after median filtering for:

[0117]

[0118] This algorithm effectively removes impulse interference such as salt-and-pepper noise by sorting the pixel values ​​in the neighborhood and taking the median value.

[0119] Next, histogram equalization is performed to enhance image contrast. Let the image have 100 gray levels. The probability of the grayscale distribution of the original image is , For the first One gray level, gray levels after histogram equalization It can be calculated using the following formula:

[0120] in, It is grayscale. The number of pixels that appear This represents the total number of pixels in the image. This transformation stretches the grayscale distribution of the image, making image details clearer and facilitating subsequent feature extraction.

[0121] 3. Multi-sensor data time synchronization and format conversion

[0122] Time synchronization is achieved using the timestamp information inherent in the data from each sensor. Let the timestamp sequence of the camera data be... The IMU data timestamp sequence is The timestamp sequence of the lidar data is A synchronization method based on minimum time difference is adopted for a given moment. Find with The sensor data with the smallest time difference, i.e.:

[0123] The data from multiple sensors at corresponding times are matched to ensure data consistency over time.

[0124] For 3D point cloud data acquired by lidar To facilitate subsequent processing, it is converted into a voxel mesh representation. Let the resolution of the voxel mesh be... Then point cloud data Points in Voxel coordinates mapped to a voxel grid The calculation formula is:

[0125] ;

[0126] in, These are the minimum coordinate values ​​for the point cloud data. Voxelization reduces the data volume while preserving the spatial structure information of the environment, thus improving data processing efficiency.

[0127] In the multimodal data acquisition and preprocessing section, addressing the challenges of data susceptibility to interference and the difficulty of multi-source data fusion under dynamic occlusion environments, this section utilizes innovative sensor selection and combination, refined image preprocessing algorithms, and efficient time synchronization and format conversion strategies to provide a high-quality, unified, and complementary data foundation for subsequent dynamic occlusion perception and cross-modal alignment, thereby enhancing the agent's perception capabilities in complex environments.

[0128] 2. Dynamic Occlusion Perception and Prediction: A dynamic occlusion perception model is constructed using deep learning algorithms to analyze visual images and identify information such as the type, location, and motion state of occluders. Based on historical data and current observations, a spatiotemporal sequence-based prediction algorithm is used to predict the motion trajectory and occlusion range changes of occluders over a future period, thus proactively identifying potential occlusion risks.

[0129] (II) Dynamic Occlusion Sensing and Prediction

[0130] 1. Construction of Dynamic Occlusion Perception Model

[0131] To achieve effective perception of dynamic occlusions, a dynamic occlusion perception model integrating a convolutional neural network (CNN) and a long short-term memory network (LSTM) is constructed. The input is a preprocessed sequence of RGB images. Where T represents the number of frames in the image sequence, , , These are the width and height in pixels, respectively.

[0132] First, using ResNet-50 as the backbone CNN network, for each frame of image... Feature extraction is performed; ResNet-50 addresses the gradient vanishing and degradation problems in deep network training through residual structures, and its output feature maps... ,in , The spatial dimensions of the feature map. denoted as the number of channels. During feature extraction, the network learns local and global visual features in the image, such as edges, textures, and object contours, through convolution operations. The convolution operation can be represented as:

[0133] ;

[0134] in, For the first Feature map of the layer For convolution kernel weights, For bias terms, This represents the convolution operation. This is an activation function (such as the ReLU function).

[0135] Then, the continuous Feature map sequence of frames The input is fed into the LSTM network. The LSTM network uses a gating mechanism (input gate) to control the input. Forgotten Gate Output gate LSTM selectively memorizes and forgets information, effectively capturing the movement patterns and time-series characteristics of occluded objects. The computation process of LSTM is as follows:

[0136]

[0137]

[0138]

[0139]

[0140]

[0141] in, For LSTM The hidden state at all times In cellular state, This indicates element-wise multiplication. This is the weight matrix. This is the bias vector.

[0142] Ultimately, the LSTM network outputs the type of obstruction. Location coordinates and speed of movement Information such as these is used to make predictions through classification and regression layers. The loss function is a weighted sum of cross-entropy loss (for type classification) and mean squared error loss (for position and velocity regression).

[0143]

[0144] in, For classifying losses, To regress the loss, The weighting coefficients are used to optimize the loss function through training, enabling the model to accurately identify information related to occlusions.

[0145] 2. Spatiotemporal sequence prediction based on Transformer

[0146] To predict the future trajectory and occlusion range changes of occluding objects, a Transformer-based spatiotemporal sequence prediction algorithm is employed. This algorithm uses historical data... The position and motion information of the occluders in the frame are used as the input sequence. ,in ,Include Position and velocity information at any given time.

[0147] The Transformer model consists of a multi-head attention mechanism and a feed-forward network. The multi-head attention mechanism uses different attention heads to compute in parallel, capturing the dependencies between different positions in the input sequence. Its computation process is as follows:

[0148]

[0149]

[0150]

[0151] in, , , These are query, key, and value matrices, respectively. For the dimension of the key, For the number of attention heads, , , and This is the weight matrix.

[0152] After multi-layer processing using a multi-head attention mechanism and a feedforward neural network, the model outputs the predicted positions of the occluded objects at the next (N) time steps. and prediction speed During the prediction process, the loss function is to minimize the mean squared error between the predicted and actual values.

[0153]

[0154] By training and optimizing the loss function, the model can accurately predict the future dynamic changes of occluders, providing key information for subsequent cross-modal spatiotemporal alignment optimization.

[0155] This section innovatively integrates CNN, LSTM, and Transformer to construct a complete system from dynamic occlusion perception to future change prediction. Addressing the characteristics of occlusion in dynamic occlusion environments that are variable and difficult to predict, it utilizes the powerful feature learning and sequence modeling capabilities of deep learning to achieve high-precision occlusion state recognition and dynamic prediction, laying the foundation for the accurate operation of visual language navigation in complex environments.

[0156] 3. Cross-modal feature extraction and fusion: Features are extracted from visual images and language instructions separately. For visual images, convolutional neural networks (CNNs) are used to extract local and global visual features; for language instructions, natural language processing techniques, such as recurrent neural networks (RNNs) or Transformer models, are used to extract semantic features. Then, the extracted visual and semantic features are fused to construct a cross-modal joint feature space, enhancing the joint expressive power of the environment and instructions.

[0157] (III) Cross-modal feature extraction and fusion

[0158] 1. Visual Feature Extraction

[0159] To address the issues of visual information being easily interfered with and having incomplete features under dynamic occlusion environments, an improved dual-branch CNN structure is employed for visual feature extraction. The input is a preprocessed RGB image. ,in and These represent the width and height of the image in pixels, respectively.

[0160] The main branch uses a lightweighted EfficientNet-B5 as its backbone network. This network optimizes its depth, width, and resolution through a composite scaling method, reducing computational cost while maintaining accuracy. Its convolution operation process can be represented as:

[0161] in, This is the feature map of the l-th layer. For convolution kernel weights, For bias terms, This represents the convolution operation. The activation function is used (such as the Swish function). The main branch primarily learns global semantic and overall structural features in the image.

[0162] The auxiliary branch uses ResNet-34 with dilated convolutions. Dilated convolutions expand the receptive field without increasing the number of parameters by inserting holes into the standard convolution kernel, thus better capturing the contextual information of the occluded parts. Its convolution operation is as follows:

[0163]

[0164] in, This indicates a dilated convolution operation. and Each is a secondary branch. Layer weights and biases.

[0165] Feature maps output from the main branch and auxiliary branches and Channel stitching is performed to obtain the fused visual feature map. Then, through the Global Average Pooling (GAP) operation, the feature map is compressed into a fixed-length global visual feature vector. ,Right now:

[0166] .

[0167] 2. Semantic Feature Extraction

[0168] To accurately extract semantic information from natural language instructions, a Transformer model based on a dynamic attention mechanism is designed. The input natural language instruction is a sequence of words. ,in The number of words in the instruction. For the first The word vectors of each word are obtained by using a pre-trained Word2Vec model to obtain the initial word vectors.

[0169] In the Transformer's encoding layer, the word vectors are first positionally encoded, incorporating positional information into the word vectors. The formula for calculating the positionally encoded vector \(PE_{pos}\) is:

[0170]

[0171]

[0172] in, For word position, For dimensional indexing, The dimension is the word vector.

[0173] The word vectors, after positional encoding, are input into a multi-head attention mechanism and a feed-forward network. In the multi-head attention mechanism, a dynamic attention weight adjustment strategy is introduced, dynamically adjusting the attention weights based on the relevance of keywords in the instructions to the navigation task. Let the attention weights be... It is calculated using the following formula:

[0174]

[0175]

[0176] in, , These are the query and key vectors, respectively. For the dimension of the key, For words and Relevance score to navigation task These are the weighting coefficients.

[0177] After multiple layers of encoding, the average value of the last hidden state is taken as the semantic feature vector. .

[0178] 3. Feature fusion

[0179] A feature fusion network based on a gating mechanism is used to process visual feature vectors. and semantic feature vectors The two feature vectors are first concatenated to obtain...

[0180] .

[0181] Then, the fusion weights of visual and semantic features are calculated using a gating unit. The output of the gating unit is shown below. The calculation formula is:

[0182] in, This is the weight matrix. For bias vectors, This is the Sigmoid activation function.

[0183] Final cross-modal joint feature vector Calculated using the following formula:

[0184]

[0185] in, This indicates element-wise multiplication. This joint feature vector integrates visual and semantic information, better adapting to the needs of visual-language navigation in dynamic occlusion environments, and providing rich and effective feature representations for subsequent cross-modal spatiotemporal alignment optimization.

[0186] The above-mentioned content, through an innovative dual-branch CNN structure, dynamic attention Transformer model, and gated feature fusion method, addresses the problems of incomplete visual features and easy bias in semantic understanding under dynamic occlusion environment. It achieves high-precision cross-modal feature extraction and fusion, effectively enhances the agent's ability to jointly express the environment and instructions, and provides core support for accurate visual language navigation.

[0187] Cross-modal spatiotemporal alignment optimization: Based on the prediction results of dynamic occlusion, visual features are compensated and adjusted in time and space. In the temporal dimension, visual features that may be occluded in the future are extracted and stored in advance according to the movement speed and direction of the occluding object. In the spatial dimension, geometric transformation and image inpainting techniques are used to restore the visual information of the occluded parts. At the same time, the alignment relationship between visual features and semantic features is optimized by combining the semantic information of language instructions, ensuring that visual observation and language instructions are consistent in time and space under dynamic occlusion environment.

[0188] (iv) Cross-modal spatiotemporal alignment optimization

[0189] 1. Time dimension compensation

[0190] To address the temporal discontinuity of visual features caused by dynamic occlusion, a temporal dimension compensation mechanism is designed based on the dynamic occlusion prediction results. The dynamic occlusion prediction model outputs future... Predicted location of the occluder at time 1 and prediction speed ,in This refers to the current moment.

[0191] For regions that may be occluded in the future, feature extraction is performed in advance in the current frame. Define occlusion risk regions. When predictions are made in When a certain region is occluded, the set of pixels within that region can be represented as: At the current moment ,right For the region, a feature pre-extraction method based on a sliding window is used. Let the sliding window size be \(w \times h\), with a step size of... exist Slide within the region to extract features from the image patch within each window.

[0192] For the Image blocks within a sliding window The feature vector is obtained by using a two-branch CNN model built in the visual feature extraction part to extract features. Store these feature vectors in the feature cache. In, that is , This represents the number of sliding windows.

[0193] When the actual moment of occlusion occurs At that time, directly from the feature cache The feature vector of the corresponding region is obtained and used for subsequent cross-modal processing to avoid feature loss due to occlusion. The specific acquisition process can be represented as follows:

[0194]

[0195] in, The function is based on the currently occluded area. From feature cache Retrieve the corresponding feature vector from the middle.

[0196] 2. Spatial Dimension Restoration

[0197] For the occluded regions, an image inpainting technique based on Generative Adversarial Networks (GANs) and geometric constraints is used to restore spatial dimensions. Let the occluded image be... The mask is , The area with a median value of 1 represents the occluded part.

[0198] Generative adversarial networks consist of a generator and discriminator Composition. Generator The goal is to generate reasonable image content for the occluded region based on information from the unoccluded region. Its input is a concatenated tensor containing features of the unoccluded region and the occlusion mask. ,in The image shows the unoccluded area. The generator's output is the repaired image. .

[0199] Discriminator This is used to determine whether the input image is a real, unoccluded image or a repaired image generated by the generator. Its output is a probability value representing the probability that the input image is a real image. .

[0200] To improve the accuracy of the repaired image, geometric constraints are introduced. Based on the geometric shape and position information of the occluder obtained from dynamic occlusion prediction, geometric transformation constraints are applied to the image generated by the generator. Let the predicted geometric transformation parameters of the occluder be... , representing rotation, translation, and scaling parameters, respectively. For the generated repaired image Perform geometric transformations:

[0201]

[0202] At the same time, the design includes measures to combat losses. Perceived loss and geometric constraint loss Joint loss function:

[0203]

[0204] in,

[0205]

[0206]

[0207] Features extracted from a pre-trained VGG network, , , These are the weighting coefficients. The joint loss function is optimized through training to make the generated repaired image more visually and geometrically similar to the real image.

[0208] 3. Alignment optimization

[0209] After completing temporal and spatial dimension compensation and restoration, the alignment relationship between visual and semantic features is optimized. The restored and compensated visual feature vectors are then compared. and semantic feature vectors The input is fed into an attention-based alignment optimization network.

[0210] Calculate the attention weights between visual and semantic features. :

[0211]

[0212] in, and The first and second parts of the visual feature vector and semantic feature vector are respectively the first and second parts of the semantic feature vector. and the Each element.

[0213] The visual features are weighted and summed according to the attention weights to obtain a visual feature vector aligned with the semantic features. :

[0214] By minimizing the mean squared error between the aligned visual feature vector and the semantic feature vector:

[0215]

[0216] Optimizing the alignment relationship enables more precise spatiotemporal alignment between visual observations and language commands, providing a reliable feature base for subsequent path planning and navigation control.

[0217] This section addresses the spatiotemporal alignment challenge of visual language navigation in dynamic occlusion environments by employing innovative temporal dimension compensation strategies, spatial dimension recovery methods based on GANs and geometric constraints, and attention-driven alignment relationship optimization. It achieves comprehensive and high-precision cross-modal spatiotemporal alignment optimization, effectively improving the navigation accuracy and robustness of intelligent agents in complex dynamic environments.

[0218] 5. Path planning and navigation control:

[0219] Based on the optimized cross-modal alignment results, reinforcement learning or traditional path planning algorithms are used to plan the optimal path from the current position to the target position in the environmental map. The agent navigates according to the planned path and dynamically adjusts the path in real time based on new environmental perception information and command requirements to ensure the successful completion of the navigation task.

[0220] (v) Path planning and navigation control

[0221] 1. Construction of a path planning algorithm based on hybrid reinforcement learning

[0222] To achieve efficient path planning in dynamic occlusion environments, a hybrid reinforcement learning path planning algorithm based on Deep Q-Network-Space (DQN-Space) and Fast Exploratory Random Tree (RRT*) is constructed. The optimized cross-modal joint feature vector is used... Current position coordinates of the agent and target position coordinates As input to the algorithm.

[0223] In the DQN - Space section, the state space is defined. Its elements include agent location and environmental features (through cross-modal joint feature vectors). This includes information such as representation, dynamic occlusion prediction (e.g., the future position and velocity of the occluding object), etc. Action space This includes the agent's movement actions, such as discrete actions like moving forward, turning left, turning right, accelerating, and decelerating, denoted as... , This represents the number of actions.

[0224] Q-value function Indicates the state Next action The expected cumulative rewards, of which These are the network parameters. A deep neural network (DNN) is used to approximate the Q-value function. The network input is state information, and the output is the Q-value corresponding to each action. During training, the temporal difference (TD) learning algorithm is used to update the network parameters, and the loss function is... Defined as:

[0225]

[0226] The target value \(y\) is calculated using the following formula:

[0227] To perform the action The instant reward obtained afterward Discount factor Used to balance immediate rewards and future rewards. To perform the action The next state after that, For target network parameters (periodically from the current network) (Copy and update).

[0228] RRT* is partly used for quickly exploring feasible paths in complex environments. (Based on the agent's current position...) As the root node, random sampling points are used in the environment map. Find the distance in the tree The nearest node ,along arrive Direction by step Expand new nodes During the expansion process, based on the dynamic occlusion prediction results, it is determined whether the new node is in a safe area (i.e., not occupied by predicted occupants). If safe, the new node is added to the tree, and the path is optimized through rewiring operations to reduce path cost. Path cost function. Defined as:

[0229]

[0230] in, A sequence of path nodes. For nodes and The Euclidean distance between them To obscure the penalty weight, This represents the number of times the path passes through the predicted occluded region.

[0231] In the hybrid strategy, during the initial navigation phase, the RRT* algorithm is used to quickly generate initial feasible paths, providing exploration guidance for DQN-Space. As training progresses, the decision weights of DQN-Space are gradually increased, enabling it to perform better path planning based on the learned strategy, ultimately obtaining the optimal path from the current position to the target position. .

[0232] 2. Dynamic navigation control and path adjustment

[0233] The agent follows the planned path Navigation is performed by dynamically detecting new dynamic occlusions on the path through real-time acquisition of environmental information (such as visual images, LiDAR point clouds, etc.) and cross-modal alignment results. Let the agent's current time be... The real-time sensing of the location information of the obstruction is ,like With path There is an intersection, that is , ( If the distance threshold is reached, the path adjustment mechanism will be triggered.

[0234] During path adjustment, the dynamic occlusion perception and prediction steps are first re-executed to obtain new occlusion information. Then, starting from the current agent position and keeping the target position unchanged, the path planning algorithm based on hybrid reinforcement learning is run again to generate a new path. At the same time, by combining semantic feature information, it is ensured that the new path conforms to the requirements of the language instructions. For example, if the language instruction requires "avoiding the obstacle on the left", then the penalty weight of the left area is increased during path planning.

[0235] During navigation, the agent's motion control employs a proportional-integral-derivative (PID) controller. Let the desired direction of travel for the agent be... The actual driving direction is The speed deviation is ,in and These represent the desired speed and the actual speed, respectively. The output control quantity of the PID controller. for:

[0236]

[0237] in, , , The proportional, integral, and derivative coefficients for angle control. , , These are the proportional, integral, and derivative coefficients for speed control. By adjusting these coefficients, the agent can stably travel along the planned path and respond promptly to environmental changes.

[0238] This section employs an innovative hybrid reinforcement learning path planning algorithm and dynamic navigation control strategy, closely integrating the visual-language navigation requirements under dynamic occlusion environments. This achieves efficient path planning and precise navigation control, enabling the agent to flexibly cope with dynamic occlusion in complex and ever-changing environments and successfully complete navigation tasks, significantly improving the system's practicality and reliability.

[0239] Example: Multimodal Data Acquisition and Preprocessing

[0240] (a) Sensor Deployment

[0241] 1. Install a device with a resolution of [resolution value] at the center of the top of the agent. Frame rate The RGB camera ensures a horizontal viewing angle of [missing information]. Vertical viewing angle reaches To cover the main field of view in front of and to the sides of the intelligent agent.

[0242] 2. The IMU sensor, integrating a three-axis accelerometer and a three-axis gyroscope, is fixed at the center of gravity of the smart body. Its acceleration measurement accuracy is... The accuracy of angular velocity measurement is It outputs the acceleration and angular velocity data of the intelligent agent in real time through the I²C communication protocol.

[0243] 3. Install a 16-line 3D LiDAR in front of the intelligent agent, with a scanning frequency of 10Hz and an angular resolution of [missing information]. The scanning range is horizontal. ,vertical ( to ), to obtain three-dimensional spatial information of the environment in the form of point clouds.

[0244] (II) Visual Image Preprocessing

[0245] 1. Perform median filtering on an RGB image: Set A sliding window of varying size sorts the values ​​of each pixel in the image and its nine neighboring pixels, using the median as the output value for that pixel, thus removing salt-and-pepper noise.

[0246] 2. Perform histogram equalization: Calculate the percentage of pixels at each gray level in the image, and map the original image gray levels to new gray levels according to the cumulative distribution function, thereby enhancing image contrast. For example, for gray levels... Its new grayscale level Through formula Calculation, where grayscale The percentage of pixels.

[0247] (III) Multi-sensor data synchronization and format conversion

[0248] 1. Time Synchronization: Record the timestamps of data from each sensor and synchronize the data using linear interpolation. Using camera data as a reference, for IMU and LiDAR data, find the two time points closest to the reference timestamp, and obtain the data value at the synchronization moment through linear calculation.

[0249] 2. LiDAR point cloud data format conversion: Convert the point cloud data to a resolution of [resolution value missing]. A voxel mesh. For each point cloud coordinate... Calculate the corresponding voxel grid coordinates. The formula is , , ,in The minimum coordinate value of the point cloud data.

[0250] II. Dynamic Occlusion Perception and Prediction

[0251] (I) Training and Operation of Dynamic Occlusion Perception Model

[0252] 1. Model Training: Prepare a dataset containing 100,000 images of dynamic occlusion in different scenes, labeled with occlusion type, location, and motion information. Divide the dataset into... The dataset is divided into training, validation, and test sets. During training, the Adam optimizer is used, and the learning rate is initialized to [value missing]. Decrease every 10 epochs Set the weighting coefficients for classification loss and regression loss. Train for 50 epochs.

[0253] 2. Real-time perception: The preprocessed RGB image sequence (5 consecutive frames per input) is input into the trained model. The ResNet-50 backbone network extracts image features, and the LSTM network analyzes the feature sequence, outputting the type (pedestrian, vehicle, etc.), location coordinates, and motion speed of occluders in the current frame.

[0254] (II) Spatiotemporal sequence prediction based on Transformer

[0255] 1. Data preparation: Collect the position and speed information of the obstruction within the past 10 seconds, record it once every 0.1 seconds, and form an input sequence of length 100.

[0256] 2. Model Training: A 6-layer Transformer encoder with 8 attention heads and a hidden layer dimension of 512 was used. The mean squared error loss function, Adam optimizer, and learning rate of 0.0001 were employed for 30 epochs.

[0257] 3. Prediction Execution: Input the occlusion information of the past 10 seconds into the trained model, and output the predicted position and velocity of the occlusion within the next 5 seconds (each time point is 0.1 seconds), providing a basis for subsequent processing.

[0258] III. Cross-modal feature extraction and fusion

[0259] (a) Visual Feature Extraction

[0260] 1. Dual-branch CNN network construction: The main branch uses a lightweight EfficientNet-B5, and the auxiliary branch uses ResNet-34 with dilated convolutions. In the main branch, the kernel size of the convolutional layers is 3×3, and the stride is 1; the dilated convolutional layers in the auxiliary branch have dilation rates of 1, 2, 4, and 8, respectively.

[0261] 2. Feature Extraction: The preprocessed RGB image is input into a dual-branch network. The main branch extracts global semantic features, while the auxiliary branch extracts local detail features. The feature maps output from the two branches are concatenated by channel and then subjected to global average pooling to obtain a global visual feature vector of length 1024.

[0262] (II) Semantic Feature Extraction

[0263] 1. Construct a Transformer model based on dynamic attention mechanism: it contains 6 encoder layers, 768 hidden layer dimensions, and 12 attention heads.

[0264] 2. Data Processing: After segmenting the natural language instructions, word vectors are obtained using a pre-trained Word2Vec model, and positional encoding is added. When calculating the relevance score between words, cosine similarity is used to measure the degree of association between words and navigation task keywords (such as "target" and "avoid").

[0265] 3. Feature extraction: Input the processed instructions into the Transformer model, take the average value of the last hidden state, and obtain a semantic feature vector of length 768.

[0266] (III) Feature Fusion

[0267] 1. Concatenate the visual feature vector and the semantic feature vector into a vector of length 1792, and input it into the gating unit.

[0268] 2. The gating unit consists of a fully connected layer with an output dimension of 1024 and an activation function of sigmoid. The gating unit calculates the fusion weights to obtain a cross-modal joint feature vector for subsequent operations.

[0269] IV. Cross-modal spatiotemporal alignment optimization

[0270] (a) Time dimension compensation

[0271] Based on the dynamic occlusion prediction results, determine the areas that may be occluded within the next 5 seconds. Specifically, for each potentially occluded region, a 32×32 sliding window is used with a step size of 16 to extract image patch features within the window. The extracted feature vectors are stored in a feature cache using a hash table structure, with region coordinates and timestamps as keys for easy and fast retrieval.

[0272] (ii) Spatial Dimension Restoration

[0273] 1. Training the Generative Adversarial Network: The generator uses a U-Net architecture, and the discriminator uses a PatchGAN architecture. Training is performed using a dataset containing 20,000 occluded images and their restoration labels. The weighting coefficients for adversarial loss, perceptual loss, and geometric constraint loss are set to [values ​​to be inserted here]. , , Train for 50 epochs.

[0274] 2. Image Restoration: For occluded images, the image of the unoccluded area and the occlusion mask are concatenated and input into the generator to obtain the restored image. Based on the geometric transformation parameters predicted by the occluder, the restored image is rotated, translated, and scaled to conform to the actual scene.

[0275] (III) Alignment Optimization

[0276] 1. Input the restored and compensated visual feature vectors and semantic feature vectors into an alignment optimization network based on an attention mechanism.

[0277] 2. Calculate the attention weights between visual and semantic features, and sum the visual features by weight to obtain the aligned visual feature vector. Optimize the alignment relationship by minimizing the mean squared error between the aligned visual feature vector and the semantic feature vector. The Adam optimizer is used during training with a learning rate of 0.001 for 10 epochs.

[0278] V. Path Planning and Navigation Control

[0279] (I) Path planning based on hybrid reinforcement learning

[0280] 1. Initialize the DQN - Space network: The number of neurons in the input layer is determined according to the dimension of the state space (approximately 2000 dimensions). There are 3 hidden layers with 1024, 512, and 256 neurons in each layer, respectively. The number of neurons in the output layer is consistent with the dimension of the action space (assuming 8 actions, i.e., 8 neurons).

[0281] 2. Training DQN - Space: Setting the discount factor The experience replay pool has a capacity of 100,000 data points. The network parameters are updated every 100 data points collected, and the target network parameters are updated every 10 epochs.

[0282] 3. Combining the RRT algorithm: Starting from the agent's current position, 1000 points are randomly sampled from the environment map, and the RRT algorithm is used to generate an initial path. The occlusion penalty weight is included in the path cost function. Set it to 5.

[0283] 4. Hybrid strategy execution: In the early stages of navigation, the path generated by the RRT algorithm accounts for 70% of the decision weight, and DQN-Space accounts for 30%. As training progresses, every 100 time steps, the RRT weight decreases by 5%, and the DQN-Space weight increases by 5%, until DQN-Space completely dominates path planning.

[0284] (ii) Dynamic navigation control and path adjustment

[0285] 1. The agent travels along the planned path, adjusting its motion via a PID controller. The proportional, integral, and derivative coefficients for angle control are set as follows: , , The proportional, integral, and derivative coefficients for speed control are respectively set as follows: , , .

[0286] 2. Real-time detection of dynamic occlusions along the path. If a new occlusion is detected within 0.5 meters of the path, path adjustment is triggered. The dynamic occlusion perception and prediction, cross-modal feature extraction and fusion, and path planning steps are re-executed to generate a new path and continue navigation.

[0287] References

[0288] [1] Sima Shuanglin, Huang Yan, He Keji, et al. Research progress in visual language navigation[J]. Acta Automatica Sinica, 2023, 49(1): 1-14. [2] YUE L, ZHOU D, XIE L, et al. Safe-vln: Collisionavoidance for vision-and-language navigation of autonomous robots operating in continuous environments[J]. IEEE Robotics and Automation Letters, 2024. [3] GENG P, LUO M, LI T, et al. Duality-Based Optimization of Occlusion Avoidance for Active Optical Navigation System in Robotic Orthopedic Surgeries[J]. IEEE Transactions on Automation Science and Engineering, 2025. [4] Feng Cheng, Zhang Congxuan, Chen Zhen, et al. Motion occlusion detection of image sequences based on optical flow and multi-scale context[J]. Acta Automatica Sinica, 2024, 50(9): 1854-1865.[5] RAJABI N, KOSECKA J. TRAVEL: Training-Free Retrieval andAlignment for Vision-and-Language Navigation[J]. arXiv preprint arXiv:2502.07306, 2025.[6] WANG J, GUAN X, SUN Z, et al. OMEGA: EfficientOcclusion-Aware Navigation for Air-Ground Robots in Dynamic Environments viaState Space Model[J]. IEEE Robotics and Automation Letters, 2024.[7] LI Z,ZHOU G, HONG H, et al.Ground-level Viewpoint Vision-and-Language Navigationin Continuous Environments[J]. CoRR, 2025.[8] XIE Y, OU B, MA F, et al.Observation-Graph Interaction and Key-Detail Guidance for Vision and LanguageNavigation[J]. arXiv preprint arXiv:2503.11006, 2025.[9] ZHOU D, DENG J, PANGZ, et al. Exploring Graph-aware Reasoning and Bidirectional Selection forVision-Language Navigation[C] / / ICASSP 2025-2025 IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025: 1-5.

[10] HONG H, WANG S, HUANG Z, WU Q, LIU J. Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environments[C] / / Proceedings of the32nd ACM International Conference on Multimedia (MM '24). New York, NY, USA:Association for Computing Machinery, 2024: 7639–7648. DOI:10.1145 / 3664647.3681640.

[11] PASCOE G, MADDERN W, TANNER M, et al.Nid-slam: Robustmonocular slam using normalized information distance[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 1435-1444.

[12] FIROOZI R, MIR A, CAMPS GS, et al. OA-MPC: Occlusion-Aware MPC forGuaranteed Safe Robot Navigation With Unseen Dynamic Obstacles[J]. IEEETransactions on Control Systems Technology, 2024.

[13] ANDERSON P, WU Q, TENEYD, et al. Vision-and-Language Navigation: Interpreting Visually-GroundedNavigation Instructions in Real Environments[C] / / Proceedings of the IEEEConference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3674–3683.

[14] WANG Z, LEE S, LEE G H. Dynam3D: Dynamic Layered 3DTokens Empower VLM for Vision-and-Language Navigation[J]. arXiv preprintarXiv:2505.11383, 2025.

[0289] Although the present invention has been described above, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many modifications under the guidance of the present invention without departing from the spirit of the present invention, and these modifications are all within the protection scope of the present invention.

Claims

1. A visual language navigation system for cross-modal alignment in dynamic occlusion environment, characterized in that, The system comprises a multi-modal data acquisition and preprocessing module, a dynamic occlusion perception and prediction module, a cross-modal feature extraction and fusion module, a cross-modal spatio-temporal alignment optimization module, and a path planning and navigation control module, wherein: The multi-modal data acquisition and preprocessing module: uses sensors to collect visual images, motion posture and spatial distance of the environment where the agent is located, wherein: the visual image adopts a preprocessing strategy combining median filtering and histogram equalization to remove noise and enhance image contrast; in terms of multi-sensor data fusion, a unique time synchronization algorithm and point cloud data collected by the laser radar are converted into a voxel grid format; The dynamic occlusion perception and prediction module: constructs a dynamic occlusion perception model through a deep learning algorithm to identify the type, position and motion state of the occlusion, and then uses a spatio-temporal sequence prediction algorithm to predict the future motion trajectory and occlusion range change of the occlusion according to historical and current observations; The cross-modal feature extraction and fusion module: extracts features from visual images and language instructions respectively, wherein: the visual image adopts a double-branch convolutional neural network to extract features, and the language instruction extracts semantic features through a Transformer model based on a dynamic attention mechanism, and then a gate-controlled feature fusion network is used to fuse the features of the two to obtain a cross-modal joint feature vector; The cross-modal spatio-temporal alignment optimization module: obtains dynamic occlusion prediction results based on the dynamic occlusion perception model, extracts visual features of the future possible occluded area in the time dimension in advance, repairs the occluded area image using a generative adversarial network and geometric constraints in the spatial dimension, and optimizes the alignment relationship between the visual features and the semantic features through an attention mechanism; The path planning and navigation control module: uses a hybrid reinforcement learning algorithm based on a deep Q network - space and a rapid exploration random tree, combines the optimized cross-modal alignment results to plan a path, and the agent drives according to the planned path and dynamically adjusts the path according to the real-time perception of the dynamic occlusion.

2. The visual language navigation system that aligns cross-modal in dynamic occlusion environment of claim 1, wherein, In the multi-modal data acquisition and preprocessing, the RGB image collected by the visual sensor is subjected to median filtering denoising and histogram equalization enhancement, and the point cloud data collected by the laser radar is converted into a voxel grid format.

3. The visual language navigation system that aligns cross-modal in dynamic occlusion environment of claim 1, wherein, The dynamic occlusion perception and prediction module constructs a dynamic occlusion perception model through a deep learning algorithm to identify the type, position and motion state of the occlusion, and then uses a spatio-temporal sequence prediction algorithm to predict the future motion trajectory and occlusion range change process of the occlusion according to historical and current observations; comprising: 101 Construct a dynamic occlusion perception model that fuses convolutional neural networks and long short-term memory networks: a pre-processed RGB image sequence where T denotes the number of frames of the image sequence, , , respectively the width and height in pixels of the image. ResNet-50 is used as the backbone convolutional neural network to extract features from each frame of image ResNet-50 solves the gradient vanishing and degradation in deep network training through residual structure and outputs feature maps wherein , is the spatial size of the feature map, is the number of channels; In the feature extraction process, the edges, textures, object contour local and global visual features in the image are learned through convolution operations, which can be represented as: ； wherein, is the first feature map of the layer, is the convolution kernel weight, is the bias term, denotes the convolution operation, is the activation function; Then, the continuous sequence of feature maps of the frame ; the long short-term memory network selects information to remember and forget selectively through a gating mechanism, and the motion law and time sequence characteristics of the occlusion are captured. The calculation process of the long short-term memory network is as follows: ； ； ； ； ； wherein, is a long short-term memory network in the hidden state at time step t, is a cell state, denotes element-wise multiplication, is a weight matrix, is a bias vector Finally, the long short-term memory network outputs the type of the occluder , the position coordinates , and the motion velocity information, which realizes the prediction through the classification and regression layers, and the loss function thereof adopts the weighted sum of the cross-entropy loss and the mean square error loss: ； wherein, is a classification loss, is a regression loss, is a weighting coefficient, the loss function is optimized by training to enable the model to accurately identify the occlusion-related information; 102 Spatio-temporal sequence prediction based on Transformer With history Occluder position and motion information of frames as input sequence Wherein , comprising Position and velocity information at time instant; The multi-head attention mechanism calculates in parallel through different attention heads to capture the dependency between different positions in the input sequence, and the calculation process is as follows: ； ； ； wherein, , , are the query, key and value matrices, respectively, is the dimension of the keys, is the number of attention heads, , , and are weight matrices; After multi-head attention mechanism and multi-layer processing of the feedforward neural network, the dynamic occlusion perception model outputs the predicted positions of the occluders at the next N time points and the predicted speeds ​ In the prediction process, the mean square error between the predicted value and the true value is minimized as the loss function: ； The loss function is optimized by training to enable the dynamic occlusion perception model to accurately predict the future dynamic changes of the occluder, providing key information for subsequent cross-modal spatio-temporal alignment optimization.

4. The visual language navigation system for cross-modal alignment in dynamic occluded environments of claim 1, wherein, The cross-modal feature extraction and fusion module adopts a main branch of a double-branch convolutional neural network, which uses a lightweight EfficientNet to extract global semantic features, and an auxiliary branch which uses a ResNet containing a hollow convolution to extract local detail features. The two feature maps are spliced and then subjected to global average pooling to obtain a visual feature vector process; including: The input is a pre-processed RGB image wherein and are the width and height in pixels of the image, respectively. The main branch uses a lightweight EfficientNet-B5 as the backbone network, and its convolution operation process can be represented as: ； wherein, is a feature map of the lth layer, is a convolution kernel weight, is a bias term, denotes a convolution operation, is an activation function; The auxiliary branch uses a ResNet-34 with a hollow convolution structure. The hollow convolution expands the receptive field without increasing the number of parameters by inserting a hollow in the standard convolution kernel, which can better capture the context information of the occluded part. Its convolution operation is: ； wherein, denotes a dilated convolution operation, and are the weights and bias of the auxiliary branch’s layer, respectively. The feature maps output by the main branch and the auxiliary branch And Channel splicing is performed to obtain a fused visual feature map After a global average pooling operation, the feature map is compressed into a fixed-length global visual feature vector That is, 。 5. The visual language navigation system that aligns across modalities in dynamic occluded environments of claim 4, wherein, In the cross-modal feature extraction and fusion, the Transformer model based on the dynamic attention mechanism performs word segmentation on the natural language instructions, obtains word vectors, and adds position encoding to extract semantic features. The attention mechanism dynamically adjusts the attention weight according to the relevance of the keywords in the instructions and the navigation task process; including: The input natural language instruction is a word sequence wherein: is the number of instruction words, is a word vector representation of the th word, the initial word vector is obtained by a pre-trained Word2Vec model; In the encoding layer of the Transformer, the word vector is first subjected to position encoding, which integrates the position information into the word vector. The calculation formula of the position encoding vector \(PE_{pos}\) is: ； ； wherein, is a word position, is a dimension index, is a word vector dimension; The position-coded word vector is input into a multi-head attention mechanism and a feedforward neural network, a dynamic attention weight adjustment strategy is introduced, and the attention weight is dynamically adjusted according to the relevance of the keywords in the instruction and the navigation task; the attention weight is , and is calculated by the following formula: ； ； wherein, , are query and key vectors, respectively, is the dimension of the key, is a word and is the relevance score of the word to the navigation task, is a weight coefficient; After multi-layer encoding, the average value of the last layer hidden state is taken as the semantic feature vector .

6. The visual language navigation system that aligns across modalities in dynamic occluded environments of claim 5, wherein, The feature fusion network based on the gating mechanism performs a fusion process on the visual feature vector and the semantic feature vector , and includes: First, the two feature vectors are spliced to obtain ； Then, the fusion weight of the visual feature and the semantic feature is calculated by a gating unit. The output of the gating unit is The calculation formula is: ； wherein, is a weight matrix, is a bias vector, is a Sigmoid activation function Cross-modal joint feature vector is calculated by the following equation: ； wherein, represents element multiplication. This joint feature vector combines visual and semantic information, better adapts to the needs of visual language navigation in dynamic occlusion environment, and provides rich and effective feature expression for subsequent cross-modal spatio-temporal alignment optimization.

7. The visual language navigation system for cross-modal alignment in a dynamic occlusion environment according to claim 1, characterized in that, In the cross-modal spatio-temporal alignment optimization, the time dimension compensation adopts a sliding window-based method to extract features in advance and store them for the future possible occluded areas, and the spatial dimension recovery adopts a generative adversarial network to generate occluded area images, and combines the predicted geometric transformation parameters of the occluder to adjust the generated images process; including: 110 The time dimension compensation adopts a sliding window-based method to extract features in advance and store them for the future possible occluded areas; Let the dynamic occlusion prediction model output a predicted position and a predicted velocity of an occluder at a future time where is the current time; For the regions that might be occluded in the future, feature extraction is performed in advance in the current frame; define the occlusion risk region When it is predicted that a region will be occluded at moment, the pixel set in this region can be represented as ; At the current time , the method comprises the following steps: Region, a feature pre-extraction method based on a sliding window is adopted; Let the size of the sliding window be... , with step size exist Slide within the region and extract features from the image patch within each window; for the first... Image blocks within a sliding window The feature vector is obtained by using a two-branch CNN model built in the visual feature extraction part to extract features. Store these feature vectors in the feature cache. In, that is , This represents the number of sliding windows; When the actual entering occluded moment , the feature vector of the corresponding region is directly obtained from the feature cache for subsequent cross-modal processing, avoiding feature loss due to occlusion; the specific acquisition process can be represented as: ； wherein, the function according to the current occluded region retrieving the corresponding feature vector from the feature cache; retrieving the corresponding feature vector from the feature cache; 111 The spatial dimension recovery adopts a generative adversarial network to generate occluded area images; For the regions that have been occluded, the image inpainting technique based on the generative adversarial network (GAN) and geometric constraints is adopted to recover the spatial dimension. Let the occluded image be , the occlusion mask be , The region with a median value of 1 represents the occluded part; The generative adversarial network is composed of a generator and a discriminator . The target of the generator is to generate reasonable image content of the occluded area according to the information of the non-occluded area, and the input of the generator is a splicing tensor containing the features of the non-occluded area and an occlusion mask , wherein is an image of the non-occluded area, and the output of the generator is a repaired image . The discriminator is used to determine whether the input image is a real non-occluded image or a repaired image generated by the generator, and the output of the discriminator is a probability value representing the probability that the input image is a real image . According to the occluder geometry and position information obtained by dynamic occlusion prediction, the generated image is subjected to geometric transformation constraint; let the predicted geometric transformation parameters of the occluder be , respectively representing rotation, translation and scaling parameters; and the generated repaired image is subjected to geometric transformation: ​ ； At the same time, a joint loss function is designed including adversarial loss , perception loss and geometric constraint loss . ； wherein ; ； ； features extracted for a pre-trained VGG network, , , are weighting coefficients; the generated inpainting image is closer to the real situation in visual effect and geometric consistency by optimizing the joint loss function through training; After completing the time dimension compensation and spatial dimension recovery, the alignment relationship between the visual features and semantic features is optimized: the recovered and compensated visual feature vector and semantic feature vector is input into an alignment optimization network based on an attention mechanism Computing attention weights between visual features and semantic features : ； wherein, and are the first and second elements of the visual and semantic feature vectors, respectively; The visual features are weighted and summed according to the attention weights to obtain a visual feature vector aligned with the semantic features : ​ By minimizing the mean square error between the aligned visual feature vector and the semantic feature vector: 。 8. The visual language navigation system that aligns across modalities in dynamic occluded environments of claim 1, wherein, In the path planning and navigation control, the exploration random tree algorithm is used to quickly generate an initial feasible path at the beginning of navigation, and the decision weight of the Deep Q Network-Space is gradually increased to plan the optimal path as the training progresses.

9. A method for cross-modal alignment visual language navigation in dynamic occlusion environment, characterized in that, The method is implemented based on the system of any one of claims 1-8.

Images