Mobile robot visual language navigation method based on diffusion strategy

Abstract

The invention relates to a mobile robot visual language navigation method based on a diffusion strategy. The method comprises the following steps: S1, constructing an RGB and deep space-time fusion coding module; s2, constructing a natural language instruction key semantic query coding module, and effectively condensing and compressing a natural language instruction; s3, fusing vision, language, local historical displacement and global displacement information, and constructing a multi-modal condition cross attention coding module; s4, constructing a conditional diffusion Transform decoding module, obtaining a noise vector based on conditional vector decoding, acting on a multi-step denoising process, and outputting a plurality of candidate multi-step navigation trajectories; s5, constructing a navigation trajectory discriminator based on a comparative learning method, wherein the navigation trajectory discriminator is used for efficiently and autonomously discriminating the matching score of the candidate trajectory and the target point; and the like. The method has the advantages that the anti-jamming capability is high, the task completion rate is high, the method can adapt to various complex environments and dynamic changes, the generalization capability and the expandability are high, and the method can stably run in diversified application scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of visual - language multi - modal perception and decision - making for mobile robots, and particularly to a visual - language navigation method for mobile robots based on a diffusion strategy. Background Art

[0002] With the continuous progress of artificial intelligence and robotics technologies, Vision - and - Language Navigation (VLN), as a key task integrating visual perception and language understanding, has increasingly become an important direction in the research of service robots and embodied intelligence. This technology requires the robot to perceive the environment from the first - person perspective and complete complex navigation tasks according to natural language instructions. Although significant achievements have been made in end - to - end deep learning and multi - modal fusion methods in recent years, existing methods still face many challenges at the navigation strategy level. Traditional methods mostly adopt end - to - end single - step action prediction, which is prone to short - sighted effects and ignores long - term goal planning, resulting in unstable performance in dynamic or complex scenarios. At the same time, commonly used loss functions are difficult to model the diversity of navigation paths and cannot effectively handle the behavior planning problem with multiple solutions.

[0003] In recent years, diffusion models have become a powerful tool for enhancing the modeling ability of path diversity and stability due to their advantages in modeling multi - modal generation tasks. Existing research has begun to explore introducing diffusion models into sub - tasks such as Point Goal Navigation and Image Goal Navigation. The diffusion strategy can generate multi - step motion trajectories at one time, which can not only ensure the continuity of the path but also ensure the accuracy of the end point, initially forming a new paradigm of visual navigation based on the diffusion mechanism. However, for the navigation task under the joint condition of vision and language, no work has systematically constructed a diffusion model framework, and related research and technologies are still in their infancy. Summary of the Invention

[0004] Aiming at the problems existing in the prior art, the present invention proposes a visual - language navigation method for mobile robots based on a diffusion strategy. By introducing a diffusion strategy for visual - language fusion, the prediction ability of the navigation system for multi - state continuous trajectories is enhanced, and its flexibility and robustness in a changing environment are improved, thereby realizing efficient and continuous autonomous navigation.

[0005] The object of the present invention is achieved by the following technical solutions: A visual - language navigation method for mobile robots based on a diffusion strategy, the method comprising the following steps: Step S1: Construct an RGB and depth spatio - temporal fusion encoding module to achieve the alignment and fusion of RGB and depth features; Step S2: Construct a key semantic query encoding module for natural language instructions to condense and compress natural language instructions; Step S3: Construct a multi-modal conditional cross-attention encoding module to fuse visual, language, local historical displacement, and global displacement information to obtain a conditional vector, which is provided to Step S4; Step S4: Construct a conditional diffusion Transformer decoding module to decode a noise vector based on the conditional vector and apply it to a multi-step denoising process to output multiple candidate multi-step navigation trajectories; Step S5: Construct a navigation trajectory discriminator based on the contrastive learning method to autonomously discriminate the matching score between the candidate trajectory and the target point; Step S6: Select the navigation trajectory with the highest matching score, and combine the MPC trajectory tracking algorithm to achieve speed control of the mobile robot, complete the interaction with the environment and the observation update; Step S7: Repeat Steps S1 to S6 until the stop condition is met or the maximum navigation step limit is reached, and the navigation ends.

[0006] Further, in the above-mentioned Step S1, the input of the constructed RGB and depth spatio-temporal fusion encoding module is the RGB color map and depth map returned by the current robot camera. The features are respectively extracted by pre-trained models (such as CLIP and ResNet50), and the RGB and depth features are aligned and fused through a weighted learning method. Subsequently, LSTM is used to update and integrate the temporal information to obtain the moment memory feature .

[0007] Further, in the above-mentioned Step S2, the input of the constructed key semantic query encoding module for natural language instructions is the complete input string, and the features at different character positions are extracted by using a pre-trained text encoder. To reduce the computational burden and attention dispersion problems caused by long texts, the Q-Former structure is used to further extract the key semantic information. The calculation formula is as follows: where represents a natural language instruction composed of characters, represents a one-hot encoding with a dimension of , represents the feature dimension, represents the position encoding, represents the language embedding feature obtained through one-hot encoding, represents a multi-layer multi-head self-attention pre-trained encoder, represents a learnable query variable, is the query vector of length Denote a multi-layer multi-head cross-modal attention encoder. Here, let the natural language instruction feature encoding be the key vector and value vector, and be the updated text feature.

[0008] Furthermore, in the step S3, the multi-modal conditional cross-attention encoding module consists of two independent multi-layer multi-head cross-modal attentions. Let the visual feature and the text feature serve as the query vectors of each other's modalities respectively to achieve the cross-fusion of the visual and language modalities, and obtain the cross-fused visual feature and the text feature respectively; The local historical displacement information is the relative displacement of the robot from the past frame moments to the current moment , which respectively represent the relative abscissa, ordinate and heading angle offset at the moment . The global displacement information is the displacement information of the current moment relative to the starting point . These two types of displacement information are respectively encoded by independent multi-layer perceptrons to obtain displacement features and , where ; then take the first token feature in terms of length from the cross-fused visual feature and language feature, which respectively represent the global features of vision and text (represented as and respectively), and splice the above features along the feature dimension to obtain a complete conditional vector.

[0009] Furthermore, in the step S4, the query input of the conditional diffusion Transformer structure is a group of randomly sampled Gaussian noises , where represents the length of the output trajectory, and 3 represents that the output is the relative displacement prediction at each moment , the conditional feature is the conditional vector obtained in step 3, and combined with the sampling step number , training and sampling are completed based on the diffusion model principle. Due to the randomness characteristics of the diffusion model, a group of candidate multi-step navigation trajectory prediction results will be obtained.

[0010] Furthermore, in the step S5, a navigation trajectory discriminator is constructed based on the contrast learning method. The input of this discriminator is the current vision-language fusion feature and the navigation trajectory feature, which are respectively encoded by two independent encoders, and the discriminator is trained using the contrast loss function. Its formula is shown as follows: In the formula, represents the batch size (i.e., the number of input and output pairs), represents the th visual - language input fusion feature of the input, represents the th trajectory feature of the input, represents the dot - product similarity between the th visual - language input and the th predicted trajectory, represents the dot - product similarity between the th visual - language input and the th predicted trajectory, represents the dot - product similarity between the th predicted trajectory and the th visual - language input, represents the temperature vector, which is used to adjust the smoothness of the softmax.

[0011] Furthermore, the MPC trajectory tracking algorithm in step S6 optimizes the error between the target trajectory and the actual driving trajectory, and uses the model predictive control (MPC) framework to perform real - time speed adjustment to ensure that the robot runs smoothly along the selected optimal navigation trajectory and updates the observations according to environmental changes during the movement, so as to achieve efficient path tracking and dynamic adaptation.

[0012] Furthermore, in step S7, a stop action is output when one of the following two conditions is met: Condition 1: The stop probability output by the separate auxiliary stop prediction network branch is greater than the predicted stop threshold ; Condition 2: The absolute values of the actions output by the conditional diffusion model are all less than the action stop threshold ; The auxiliary stop prediction network branch receives the concatenated features of the visual global feature, text global feature, and memory feature obtained in step S3 as the input, , passes through an independent hierarchical perceptron and a Sigmoid function, and outputs the stop probability at the current moment, with a distribution range of : The auxiliary stop prediction network uses the mean - square error loss for supervision and optimization during the training process. Let represent the true - value label, and the loss function is expressed as follows: Beneficial effects Compared with the prior art, the present invention has the following advantages: (1) Strong multi-modal information fusion and anti-noise ability: By constructing a spatio-temporal fusion encoding module and a conditional diffusion Transformer decoding module, the present invention effectively fuses visual, language, historical displacement, and global displacement information, can extract high-quality feature representations in complex environments, and regresses the multi-step navigation trajectory prediction results by gradually denoising through conditional constraints, improving the accuracy and robustness of navigation.

[0013] (2) Dynamic adaptability and efficient path planning: Combining the MPC trajectory tracking algorithm, the present invention can optimize the robot speed control in real time and perform dynamic observation updates according to environmental changes. By selecting the navigation trajectory with the highest matching score and combining the contrast learning method for trajectory discrimination, the flexibility and stability of the navigation process are significantly improved, adapting to various complex environments and dynamic changes, and having strong generalization ability. Brief Description of the Drawings

[0014] Figure 1 is the overall flowchart of the method of the present invention; Figure 2 is a schematic diagram of the conditional diffusion Transformer decoding model in the method of the present invention; Figure 3 is a schematic diagram of the navigation trajectory discriminator and actuator model based on contrast learning in the method of the present invention; Figure 4 is the multi-step trajectory prediction output by the embodiment of the present invention according to the input RGB-D observation and language instruction. Detailed Embodiment

[0015] The technical solution provided by the present application will be further described below in conjunction with specific embodiments and their accompanying drawings. In combination with the following description, the advantages and features of the present application will be clearer.

[0016] A visual-language navigation method for a mobile robot based on a diffusion strategy, the overall flowchart of the method is as Figure 1 shown, and specifically includes the following steps: Step S1, construct an RGB and depth spatio-temporal fusion encoding module to achieve the alignment and fusion of RGB and depth features; The input of the visual-language navigation task is the first-person RGB image observation of the robot and a language instruction, such as "go forward and turn right at the first door to enter the living room". Among them, this method emphasizes vision and language The alignment and fusion ability, so the visual encoder can select a vision-language pre-trained encoder model such as CLIP and its variants. This model learns the semantic correspondence between images and texts through joint training on large-scale image and text datasets. By mapping images and texts to a shared embedding space, in this space, the feature vectors corresponding to semantically similar images and texts are closer. To preserve the local semantic information of the image, this method uses the feature output before the last mapping layer of this pre-trained model as the global feature representation of the color image. At the same time, the image is divided into image patches, and the maximum pooling is used to retain the features of the corresponding image patches respectively. Therefore, the extracted image features are , where represents the hidden layer dimension, and the first token position is the global feature.

[0017] Depth information is an important information source for guiding robot navigation and obstacle avoidance. Therefore, in the processing of depth images, this method can select a pre-trained network of the ResNet50 series pre-trained in the coordinate point target navigation task as the pre-trained network for depth feature extraction. This network can effectively capture the spatial structure information in the depth image and extract depth features that are helpful for positioning and obstacle avoidance.

[0018] Subsequently, a learnable multi-layer perceptron (MLP) model is used, combined with a linear layer and an activation function, to complete the alignment and fusion of RGB image features and depth information. To ensure the scale consistency after feature fusion, this method uses layer normalization (LayerNorm) to constrain the fused features, which helps the convergence and stability of the model during training, reduces the gradient explosion or disappearance problems caused by feature scale differences, and improves the training effect and final performance of the model. The calculation formula is as follows: Among them, represents the extracted RGB image features, represents the extracted depth features, represents the fused RGB-D features, , and are a set of learnable variables, represents the activation function, represents the hidden layer dimension.

[0019] Since the vision-language navigation task is a typical long-distance decision-making task, this method uses a recurrent neural network module to maintain and update historical information to help the model better understand the current navigation state. The specific calculation formula is as follows: Among them, LSTM is a classic long short-term memory network recurrent neural network structure, and the input is the visual fusion feature at the current moment and the memory feature at the previous moment , and the agent updates the memory feature every time it predicts an action .

[0020] Step S2: Construct a key semantic query encoding module for natural language instructions to condense and compress natural language instructions; Among them, to support more complex and longer natural language instructions and reduce the pressure on the computing resources of the network model caused by long sequence inputs, the present invention proposes to use the Q-Former structure to achieve the goal of condensing and compressing long-sequence natural language instructions. The condensation refers to the summary of key information, and the compression refers to the reduction of information length.

[0021] Specifically, the input of the Q-Former structure is a language instruction. First, a pre-trained text encoder (such as BERT or CLIP) is used to extract the character-level features of the complete natural language instruction , where represents the text length. On this basis, a learnable query vector with a custom length is randomly initialized , where represents the query vector length. This query vector serves as a query key and interacts and updates features with the features of the natural language instruction through a multi-head multi-layer cross-modal attention module. This process can effectively compress the information of the long-sequence instruction into representative key semantics, thereby improving the processing efficiency of the model. Through this mechanism, the network can focus on the key information in the natural language instruction, avoid wasting computing resources on redundant information, and improve the accuracy of instruction understanding at the same time. The core calculation process of this module can be represented by the following formula: Among them, represents a natural language instruction composed of characters, represents a one-hot encoding with a dimension of , represents a position encoding, represents the language embedding feature obtained through one-hot encoding, represents a multi-layer multi-head self-attention pre-trained encoder (such as BERT or CLIP, etc.), represents a multi-layer multi-head cross-modal attention encoder. Here, the natural language instruction feature encoding is used as the key vector and value vector, is the updated text feature.

[0022] Step S3: Construct a multi-modal conditional cross-attention encoding module to fuse visual, language, local historical displacement, and global displacement information to obtain a conditional vector and provide it to Step S4; Among them, the multi-modal conditional cross-attention encoding module consists of two independent multi-layer multi-head cross-modal attentions. Let the visual feature and the text feature serve as the query vectors for each other's modalities, realizing the cross-fusion of the visual and language modalities, and respectively obtaining the cross-fused visual feature and the text feature . The calculation formula is as follows: To improve the robot's ability to implicitly locate itself, local historical displacement information is used to represent the relative displacement of the robot from the past frames to the current moment, which respectively represent the relative abscissa, ordinate, and heading angle offset at time . The global displacement information is used to represent the displacement information of the current moment relative to the starting point. These two types of displacement information are respectively encoded by independent multi-layer perceptrons to obtain displacement features and . Then, the first token features in terms of length are taken from the cross-fused visual feature and language feature to respectively represent the global features of vision and text (represented as and ). These features are concatenated along the feature dimension to obtain the complete conditional vector: .

[0023] Step S4: Construct a conditional diffusion Transformer decoding module to output multiple candidate multi-step navigation trajectories by denoising the multi-step navigation trajectory noise through conditional constraint learning; Among them, the diffusion model is a generative model that transforms data into random noise by gradually adding noise (the noise addition process), and then gradually denoises through the inverse process to restore the original data. In the navigation task, the diffusion model can generate multi-step navigation trajectories with noise characteristics and meeting specific conditions to cope with complex and dynamic navigation environments. The core steps involved are as follows: Diffusion noise addition process: In the noise addition process, assume the original navigation trajectory is (including multi-step information). The diffusion process adds noise to the trajectory. After steps of iteration, the noise-added trajectory is obtained. The calculation formula for this process is as follows: Among them, is the noise coefficient of each step, which is used to control the intensity of noise addition; is the Gaussian noise added at the t-th step.

[0024] Sampling process: Restore the original navigation trajectory from the noise through the inverse diffusion process . In this process, the model guides the denoising operation of each step through the conditional feature to generate a navigation trajectory that meets the target conditions. The calculation formula for this process is: Among them, is the cumulative noise coefficient of the first t steps, is the conditional feature vector (the conditional vector obtained in step S3), is the noise estimation predicted by the network.

[0025] Specifically, the noise estimation predicted by the network is implemented by the Transformer architecture, and the structure is as Figure 2 shown, where the conditional diffusion Transformer module is stacked by the decoder structures of the Transformer for [number of layers] layers. The noise sequence is used as the query input, and the conditional feature vector is used as the key vector and value vector, participating in the calculation of the multi-head cross-attention. After iterations of the decoder for [number of layers] layers, it then passes through layer normalization (LayerNorm) and a multi-layer perceptron prediction head to output the predicted noise value.

[0026] Based on a set of random noises, through the above-mentioned noise addition and denoising processes respectively, the model can generate a set of candidate multi-step navigation prediction results.

[0027] Step S5: Construct a navigation trajectory discriminator based on the contrastive learning method to autonomously discriminate the matching score between the candidate trajectory and the target point; (as Figure 3 ) In order to select the most suitable navigation trajectory from multiple groups of candidate navigation trajectories, the present invention constructs a navigation trajectory discriminator based on the contrastive learning method. The input of this discriminator includes the current vision-language fusion feature and the features of each candidate navigation trajectory.

[0028] First, the vision-language fusion feature is subjected to feature extraction through an independent encoder, which uses the self-attention module to process the multi-modal splicing information from the global image feature, the global text feature, and the historical memory , and takes the average value of the token features as the representation. The calculation formula is as follows: The navigation trajectory features are encoded by another independent multi-layer perceptron encoder to extract the spatial structure and motion pattern of the trajectory: After the feature encoding is completed, the outputs of the two encoders will be subjected to contrastive learning to evaluate the matching score between each candidate trajectory and the target point by calculating the similarity between the vision-language features and the navigation trajectory features. The formula for calculating the matching score is , where represents the th fused vision-language input feature of the input, represents the th trajectory feature of the input, represents the dot product similarity between the vision-language input and the output trajectory.

[0029] The formula for the contrastive loss function is as follows: In the formula, represents the batch size (i.e., the number of input and output pairs), represents the temperature vector, which is used to adjust the smoothness of the softmax. The larger the value of the temperature vector, the smoother the distribution, and vice versa, the more concentrated it is. This discriminator can efficiently screen out the navigation trajectory that best matches the current environment and task, thereby improving the accuracy and execution efficiency of navigation decisions.

[0030] Step S6: Select the navigation trajectory with the highest matching score, and combine the MPC trajectory tracking algorithm to achieve speed control of the mobile robot, and complete the interaction with the environment and the observation update; As Figure 3 shown, the diffusion model outputs the relative offset distance and angle in the future steps. When applied to robot control, the MPC trajectory tracking algorithm is used to optimize the error between the target trajectory and the actual driving trajectory, and the linear velocity and angular velocity of the robot are adjusted in real time using the model predictive control framework to ensure that the robot runs smoothly along the selected optimal navigation trajectory and performs observation updates according to environmental changes during the movement, thereby achieving efficient path tracking and dynamic adaptation.

[0031] Specifically, assuming that the future step reference trajectory is obtained based on the output of the conditional diffusion model, the discrete time step is denoted as , is the linear velocity, is the angular velocity. At the t-th moment, the coordinates of the robot are , and the heading angle is , then the discrete kinematic model of the robot can be expressed as: Coordinates: Heading angle: Wherein, respectively represent the robot coordinates and heading angle at the th moment.

[0032] When the current state and the control input are given, MPC calculates the optimal control input by optimizing the cost matrix within the next T time steps, and its optimization problem is expressed as: Wherein, is the state of the system at the th moment, is the desired state (reference trajectory), is the control input, and are the weight matrices, which balance the control state and the control input. is a positive definite matrix, defined as follows: Wherein, and respectively represent the weights of the position error in the x and y directions. The larger the value, the more the system emphasizes the spatial accuracy of the trajectory. represents the weight of the angle deviation. The larger the value, the more the system emphasizes the orientation alignment.

[0033] is a positive definite matrix, defined as follows: Wherein, represents the penalty weight of the linear velocity change. The larger the value, the more conservative the system tends to move. represents the penalty weight of the angular velocity change. The larger the value, the more the system avoids sharp turning.

[0034] In the actual control process, MPC only applies the first control input and re-optimizes the control input at the next time step.

[0035] Step S7: Repeat steps S1 to S6 until the stop condition is met or the maximum navigation step limit is reached, and end the navigation.

[0036] Specifically, a stop action is output when one of the following two conditions is met: Condition 1: The stop probability output by the separate auxiliary stop prediction network branch is greater than the predicted stop threshold ; Condition 2: The absolute values of the actions output by the conditional diffusion model are all less than the action stop threshold .

[0037] The auxiliary stop prediction network branch receives the concatenated features of the visual global feature , text global feature and memory feature obtained in step S3 as input, passes through an independent hierarchical perceptron and a Sigmoid function, and outputs the stop probability at the current moment , with a distribution range of : During the training process of this auxiliary stop prediction network, mean squared error loss is used for supervision and optimization, and let represent the true value label, and the loss function is expressed as follows: The selection of the stop threshold should be debugged by selecting typical cases during testing. For Condition 1, a typical value of the predicted stop threshold can be taken as , and for Condition 2, a typical value of the action stop threshold can be taken as .

[0038] When the stop condition is met, it is considered that the instruction target has been completed, and this navigation is ended.

[0039] The method of the present invention can effectively improve the accuracy and robustness of the navigation action output of the agent based on monocular RGB-D input and language instructions. This embodiment simulates a typical scenario, and the executed natural language instruction is "Leave the current room from the front door, then turn left and enter the living room along the way. Pass by the dining table and continue to move forward towards the TV. Stop beside the coffee table and the sofa.", and its output trajectory example is as Figure 4 shown, where the first row and the second row are the RGB image and depth image obtained by the robot's first-person observation respectively, and the third row is the multi-step trajectory predicted by the diffusion model. From Figure 4As can be seen, in the initial state, the model output continuous forward signals to execute the instruction "leave the current room from the front door"; when it reaches the 64th step, the model outputs continuous left signals corresponding to the instruction "then turn left"; when it reaches the 120th step, the relative offset of the multi-step trajectory output by the model is very small, corresponding to the stop signal. This method can effectively output the local multi-step trajectory in the current situation based on language instructions for the changing RGB observations during the navigation process and finally successfully reach the correct end position. Different from the traditional method that can only output a single-step action (one of left, forward, right, stop), it improves the flexibility and continuity.

[0040] On the VLN-CE dataset (Vision-and-Language Navigation in Continuous Environments), the method of the present invention is compared with the classical methods Seq2Seq and CMA, and the results are shown in Table 1.

[0041] Table 1 Comparison of experimental results on the VLN-CE dataset Among them, TL and NE respectively represent the navigation length and the error length, in meters, and the smaller the value, the better; nDTW, OS, SR, and SPL respectively measure the trajectory fit, the arrival success rate, the stop success rate, and the path-weighted stop success rate, all in percentages, and the larger the value, the better. It can be seen that compared with the comparative methods, the method of the present invention has obvious improvements in these indicators.

[0042] The above description is only a description of the preferred embodiments of the present application and does not limit the scope of the present application in any way. Any change or modification made by any person skilled in the art according to the disclosed technical content should be regarded as an equivalent effective embodiment and fall within the scope of protection of the technical solution of the present application.

Claims

1. A mobile robot visual language navigation method based on diffusion strategy, characterized in that: The following steps are involved: Step S1: construct an RGB and deep spatiotemporal fusion encoding module to achieve alignment and fusion of RGB and deep features; Step S2: construct a natural language instruction key semantic query encoding module to condense and compress the natural language instruction; Step S3: construct a multimodal conditional cross-attention encoding module, integrate vision, language, local historical displacement and global displacement information, obtain a conditional vector, and provide it to step S4; Step S4: construct a conditional diffusion Transformer decoding module, obtain a noise vector based on conditional vector decoding, and act on the multi-step denoising process to output multiple candidate multi-step navigation trajectories; Step S5: constructing a navigation trajectory discriminator based on a contrastive learning method to autonomously discriminate the matching scores between the candidate trajectory and the target point; Step S6: Select the navigation trajectory with the highest matching score, and combine it with the MPC trajectory tracking algorithm to realize the speed control of the mobile robot, and complete the interaction with the environment and observation update; Step S7: Repeat steps S1 to S6 until the stop condition is met or the maximum number of navigation steps is reached, and navigation ends.

2. According to the diffusion strategy-based mobile robot visual language navigation method of claim 1, it is characterized in that: In step S1, the input of the constructed RGB and deep spatiotemporal fusion encoding module is the RGB color map and depth map returned by the current robot camera. The features are extracted through the pre-trained model, and the alignment and fusion of RGB and depth features are achieved through weighted learning. Then, LSTM is used to update and integrate the time series information to obtain Moment memory feature .

3. According to the diffusion strategy-based mobile robot visual language navigation method of claim 1, it is characterized in that: In the step S2, the input of the constructed natural language instruction key semantic query encoding module is the complete input string, and the pre-trained text encoder is used to extract the features of different character positions. In order to reduce the computational burden and attention dispersion problem caused by long texts, the Q-Former structure is used to further extract key semantic information. The calculation formula is as follows: in, Indicated by A natural language instruction consisting of characters. The dimension is One-hot encoding of represents the feature dimension, represents the position code, represents the language embedding feature obtained by one-hot encoding, represents a multi-layer multi-head self-attention pre-trained encoder, represents the learnable query variable, The query vector Length, represents a multi-layer multi-head cross-modal attention encoder, where the natural language instruction feature encoding are the key vector and the value vector, is the updated text feature.

4. According to the diffusion strategy-based mobile robot visual language navigation method of claim 1, it is characterized in that: In step S3, the multimodal conditional cross attention encoding module is composed of two independent multi-layer multi-head cross-modal attentions, which respectively make the visual features and text features Serving as the query vector of the other modality, realizing the cross-fusion of visual and language modalities, and obtaining the cross-fused visual features respectively With text features ; Local historical displacement information The robot is the distance from the current time Relative displacement at frame time , respectively represent the time Relative horizontal coordinate, vertical coordinate and heading angle offset; global displacement information is the displacement information of the current moment relative to the starting point ; These two types of displacement information are encoded by independent multi-layer perceptrons to obtain displacement features and ,in ; Then take the first token feature in length after cross-fusion of visual features and language features, representing the visual global features and global features of the text , concatenate the above features along the feature dimension to obtain the complete conditional vector.

5. According to the diffusion strategy-based mobile robot visual language navigation method of claim 4, it is characterized in that: In step S4, the query input of the conditional diffusion Transformer structure is a set of randomly sampled Gaussian noise ,in Represents the length of the output trajectory, and 3 represents the output of the relative displacement prediction at each moment , the conditional feature is the conditional vector obtained in step S3, combined with the sampling steps , training and sampling are completed based on the principle of diffusion model. Due to the random characteristics of the diffusion model, a set of candidate multi-step navigation trajectory prediction results will be obtained.

6. The mobile robot visual language navigation method based on diffusion strategy according to claim 5 is characterized in that: The diffusion model is a generative model that converts data into random noise through a noise-adding process of gradually adding noise, and then gradually removes noise through an inverse process to restore the original data; the specific steps are as follows: Diffusion noise adding process: In the noise adding process, assume that the original navigation trajectory is , including multi-step Information, the diffusion process will add noise to the trajectory, after Step iteration to get the trajectory after adding noise ; The calculation formula of this process is as follows: in, is the noise coefficient at each step, which controls the intensity of noise addition; is the Gaussian noise added at step t; Sampling process: From noise through inverse diffusion process Restore the original navigation track ; In this process, the model is To guide each step of the denoising operation, so as to generate a navigation trajectory that meets the target conditions; the calculation formula of this process is: in, is the cumulative noise coefficient of the first t steps, is the conditional eigenvector, is the noise estimate of the network prediction; Noise estimation of network predictions It is implemented by the Transformer architecture, where the conditional diffusion Transformer module consists of The decoder structure of the layer Transformer is stacked, the noise sequence is used as the query input, and the conditional feature vector is used as the key vector and value vector to participate in the calculation of multi-head cross-attention; After the layer decoder is iterated, it passes through the layer normalization LayerNorm and the multi-layer perceptron prediction head to output the predicted noise value.

7. The mobile robot visual language navigation method based on diffusion strategy according to claim 1 is characterized in that: In the step S5, a navigation trajectory discriminator is constructed based on the contrastive learning method. The input of the discriminator is the current visual-language fusion feature and the navigation trajectory feature. The feature encoding is completed by two independent encoders respectively. The discriminator is trained using the contrastive loss function, and its formula is as follows: In the formula, Indicates the batch size, Indicates The visual-linguistic input fusion features of input, Indicates The input trajectory features, Indicates The visual language input and The dot product similarity of the predicted trajectories, Indicates The visual language input and The dot product similarity of the predicted trajectories, Indicates The predicted trajectory and The dot product similarity of the visual language input, Represents the temperature vector, which is used to adjust the smoothness of the soft maximization.

8. The mobile robot visual language navigation method based on diffusion strategy according to claim 1 is characterized in that: The MPC trajectory tracking algorithm in step S6 optimizes the error between the target trajectory and the actual driving trajectory, and uses the model predictive control framework to perform real-time speed adjustment to ensure that the robot runs smoothly along the selected optimal navigation trajectory, and performs observation updates according to environmental changes during movement, thereby achieving efficient path tracking and dynamic adaptation.

9. The mobile robot visual language navigation method based on diffusion strategy according to claim 1 is characterized in that: In step S7, a stop action is output when one of the following two conditions is met: Case 1: The output stop probability of a separate auxiliary stop prediction network branch is greater than the predicted stop threshold ; Case 2: The absolute values ​​of actions output by the conditional diffusion model are all less than the action stop threshold ; The auxiliary stop prediction network branch receives the visual global features obtained in step S3 , global features of text and memory characteristics The splicing features As input, , after an independent layered perceptron and Sigmoid function, outputs the stopping probability at the current moment , the distribution range is : The auxiliary stop prediction network is supervised and optimized using mean square error loss during training. represents the true value label, and the loss function is expressed as follows: 。

Images