A visual language navigation method for mobile robots based on diffusion strategy

Abstract

The present invention relates to a mobile robot vision and language navigation method based on a diffusion strategy, comprising: S1, constructing an RGB and deep spatiotemporal fusion encoding module; S2, constructing a natural language instruction key semantic query encoding module to effectively condense and compress natural language instructions; S3, integrating vision, language, local historical displacement, and global displacement information to construct a multimodal conditional cross-attention encoding module; S4, constructing a conditional diffusion Transformer decoding module to obtain a noise vector based on conditional vector decoding, and applying it to a multi-step denoising process to output multiple candidate multi-step navigation trajectories; S5, constructing a navigation trajectory discriminator based on a contrastive learning method to efficiently and autonomously determine the matching scores between candidate trajectories and target points; and so on. The present invention has strong anti-interference capabilities, a high task completion rate, and can adapt to a variety of complex environments and dynamic changes. It has strong generalization and scalability and can operate stably in a variety of application scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of visual language multimodal perception and decision-making of mobile robots, and in particular to a visual language navigation method for mobile robots based on a diffusion strategy. Background Art

[0002] With the continuous advancement of artificial intelligence and robotics, vision-and-language navigation (VLN), a key task that integrates visual perception and language understanding, is becoming an increasingly important research direction in service robotics and embodied intelligence. This technology requires robots to perceive the environment from a first-person perspective and complete complex navigation tasks based on natural language instructions. Although end-to-end deep learning and multimodal fusion methods have achieved remarkable results in recent years, existing methods still face many challenges in terms of navigation strategies. Traditional methods often use end-to-end single-step action prediction, which is prone to short-sightedness and ignores long-term goal planning, resulting in unstable performance in dynamic or complex scenarios. At the same time, commonly used loss functions have difficulty modeling the diversity of navigation paths and cannot effectively handle behavior planning problems with multiple solutions.

[0003] In recent years, diffusion models, due to their advantages in modeling multimodal generation tasks, have become a powerful tool for improving path diversity and stability modeling capabilities. Research has begun exploring the application of diffusion models to subtasks such as point goal navigation and image goal navigation. Diffusion strategies can generate multi-step motion trajectories simultaneously, ensuring both path continuity and endpoint accuracy, and have initially formed a new paradigm for visual navigation based on diffusion mechanisms. However, no work has systematically constructed a diffusion model framework for navigation tasks involving combined vision and language, and related research and technology remain in their infancy. Summary of the Invention

[0004] To address the challenges of existing technologies, this paper proposes a vision-language navigation method for mobile robots based on a diffusion strategy. By incorporating a diffusion strategy based on vision-language fusion, the navigation system's ability to predict polymorphic, continuous trajectories is enhanced, improving its flexibility and robustness in changing environments, thereby enabling efficient and continuous autonomous navigation.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A mobile robot visual language navigation method based on a diffusion strategy, the method comprising the following steps:

[0007] Step S1: Construct an RGB and depth spatiotemporal fusion encoding module to achieve alignment and fusion of RGB and depth features;

[0008] Step S2: Construct a natural language instruction key semantic query encoding module to condense and compress the natural language instruction;

[0009] Step S3: Construct a multimodal conditional cross-attention encoding module to fuse vision, language, local historical displacement and global displacement information to obtain a conditional vector, which is provided to step S4;

[0010] Step S4: Construct a conditional diffusion Transformer decoding module, obtain a noise vector based on the conditional vector decoding, and act on the multi-step denoising process to output multiple candidate multi-step navigation trajectories;

[0011] Step S5: construct a navigation trajectory discriminator based on the contrastive learning method to autonomously determine the matching score between the candidate trajectory and the target point;

[0012] Step S6: Select the navigation trajectory with the highest matching score, combine it with the MPC trajectory tracking algorithm to realize the speed control of the mobile robot, and complete the interaction with the environment and observation update;

[0013] Step S7: Repeat steps S1 to S6 until the stop condition is met or the maximum number of navigation steps is reached, and navigation ends.

[0014] Furthermore, in step S1, the input of the constructed RGB and depth spatiotemporal fusion encoding module is the RGB color map and depth map returned by the current robot camera, and features are extracted through pre-trained models (such as CLIP and ResNet50), and the alignment and fusion of RGB and depth features are achieved through weighted learning. Subsequently, LSTM is used to update and integrate the temporal information to obtain Moment Memory Features .

[0015] Furthermore, in step S2, the input to the constructed natural language instruction key semantic query encoding module is the complete input string, and features at different character positions are extracted using a pre-trained text encoder. To reduce the computational burden and attention dispersion issues associated with long texts, a Q-Former structure is used to further extract key semantic information. The calculation formula is as follows:

[0016]

[0017] in, Indicated by A natural language instruction consisting of characters, Indicates the dimension One-hot encoding of represents the feature dimension, represents the position code, represents the language embedding features obtained by one-hot encoding, represents a multi-layer multi-head self-attention pre-trained encoder, represents the learnable query variable, is the query vector length, represents a multi-layer multi-head cross-modal attention encoder, where the natural language instruction feature encoding are key vectors and value vectors, is the updated text feature.

[0018] Furthermore, in step S3, the multimodal conditional cross attention encoding module is composed of two independent multi-layer multi-head cross-modal attentions, which respectively make the visual features and text features Serving as the query vector of the other modality, realizing the cross-fusion of visual and language modalities, and obtaining the cross-fused visual features respectively With text features ;

[0019] Local historical displacement information The robot is the distance from the current time Relative displacement at frame time , respectively represent the time The relative horizontal coordinate, vertical coordinate and heading angle offset of the global displacement information is the displacement information relative to the starting point at the current moment These two types of displacement information are encoded by independent multi-layer perceptrons to obtain displacement features. and ,in ; Then take the first token feature of the length after cross-fusion of visual features and language features, representing the global features of vision and text respectively (represented as and ), concatenate the above features along the feature dimension to obtain a complete conditional vector.

[0020] Furthermore, in step S4, the query input of the conditional diffusion Transformer structure is a set of randomly sampled Gaussian noise ,in Represents the length of the output trajectory, and 3 represents the output of the relative displacement prediction at each moment , the conditional feature is the conditional vector obtained in step 3, combined with the number of sampling steps , based on the principle of diffusion model, training and sampling are completed. Due to the randomness of the diffusion model, a set of candidate multi-step navigation trajectory prediction results will be obtained.

[0021] Furthermore, in step S5, a navigation trajectory discriminator is constructed based on the contrastive learning method. The input of the discriminator is the current visual-language fusion feature and the navigation trajectory feature. Feature encoding is completed by two independent encoders respectively. The discriminator is trained using the contrastive loss function, and its formula is as follows:

[0022]

[0023] Where, represents the batch size (i.e. the number of input and output pairs), Indicates the The visual-linguistic input fusion features of input, Indicates the The input trajectory features, Indicates the The visual language input and The dot product similarity of the predicted trajectories, Indicates the The visual language input and The dot product similarity of the predicted trajectories, Indicates the The predicted trajectory and The dot product similarity of the visual language input, Represents the temperature vector, which is used to adjust the smoothness of the soft maximization.

[0024] Furthermore, the MPC trajectory tracking algorithm in step S6 optimizes the error between the target trajectory and the actual driving trajectory, and uses the model predictive control (MPC) framework to perform real-time speed adjustment to ensure that the robot runs smoothly along the selected optimal navigation trajectory, and performs observation updates according to environmental changes during the movement process, thereby achieving efficient path tracking and dynamic adaptation.

[0025] Furthermore, in step S7, a stop action is output when one of the following two conditions is met:

[0026] Case 1: The output of the auxiliary stop prediction network branch is greater than the predicted stop threshold. ;

[0027] Case 2: The absolute values of the actions output by the conditional diffusion model are all less than the action stop threshold ;

[0028] The auxiliary stop prediction network branch receives the concatenation features of the visual global features, text global features and memory features obtained in step S3 As input, , after an independent layered perceptron and Sigmoid function, output the stopping probability at the current moment , the distribution range is :

[0029]

[0030] The auxiliary stop prediction network is supervised and optimized using mean square error loss during training. represents the true value label, and the loss function is expressed as follows:

[0031]

[0032] Beneficial effects

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] (1) Strong multimodal information fusion and noise resistance: The present invention effectively integrates visual, language, historical displacement and global displacement information by constructing a spatiotemporal fusion encoding module and a conditional diffusion Transformer decoding module. It can extract high-quality feature representations in complex environments and regress multi-step navigation trajectory prediction results through conditional constraint step-by-step denoising, thereby improving the accuracy and robustness of navigation.

[0035] (2) Dynamic adaptability and efficient path planning: This invention combines the MPC trajectory tracking algorithm to optimize robot speed control in real time and dynamically update observations based on environmental changes. By selecting the navigation trajectory with the highest matching score and combining it with a comparative learning method for trajectory discrimination, the flexibility and stability of the navigation process are significantly improved, adapting to a variety of complex environments and dynamic changes, and demonstrating strong generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is an overall flow chart of the method of the present invention;

[0037] Figure 2 Schematic diagram of the conditional diffusion Transformer decoding model in the method of the present invention;

[0038] Figure 3 Schematic diagram of the navigation trajectory discriminator and actuator model based on contrastive learning in the method of the present invention;

[0039] Figure 4 It is a multi-step trajectory prediction output by an embodiment of the present invention based on the input RGB-D observations and language instructions. DETAILED DESCRIPTION

[0040] The technical solution provided by this application will be further described below in conjunction with specific embodiments and accompanying drawings. The advantages and features of this application will become more apparent with reference to the following description.

[0041] A visual language navigation method for mobile robots based on diffusion strategy. The overall flow chart of this method is as follows: Figure 1 As shown, the specific steps include:

[0042] Step S1: construct an RGB and depth spatiotemporal fusion coding module to achieve alignment and fusion of RGB and depth features;

[0043] The input of the vision-language navigation task is the robot's first-person RGB image observation and language instructions, such as "go forward, turn right at the first door and enter the living room." Among them, this method emphasizes vision and language The alignment and fusion capabilities of the image are very important, so the visual encoder can choose a visual-language pre-trained encoder model such as CLIP and its variants. The model learns the semantic correspondence between images and texts through joint training on large-scale image and text datasets. The model maps images and texts into a shared embedding space, in which the feature vectors corresponding to semantically similar images and texts are close. In order to retain the local semantic information of the image, this method uses the feature output before the last mapping layer of the pre-trained model as the global feature representation of the color image, and divides the image into image blocks, and the maximum pooling is used to retain the features of the corresponding image blocks, so the image extraction features obtained are ,in Represents the hidden dimension, and the first token position is the global feature.

[0044] Depth Information Depth image processing is an important source of information for guiding robot navigation and obstacle avoidance. Therefore, this method uses the ResNet50 series model, previously trained in point target navigation tasks, as a pre-trained network for depth feature extraction. This network effectively captures the spatial structure information in depth images and extracts depth features that aid in positioning and obstacle avoidance.

[0045] Subsequently, a learnable multi-layer perceptron (MLP) model is used, combined with linear layers and activation functions, to align and fuse RGB image features and depth information. To ensure scale consistency after feature fusion, this method uses layer normalization (LayerNorm) to constrain the fused features. This helps improve the convergence and stability of the model during training, reduces gradient explosion or vanishing problems caused by feature scale differences, and improves the model's training results and final performance. The calculation formula is as follows:

[0046]

[0047] in, represents the extracted RGB image features, represents the extracted deep features, represents the fused RGB-D features, , and is a set of learnable variables, represents the activation function, represents the hidden dimension.

[0048] Since the vision-language navigation task is a typical long-distance decision-making task, this method uses a recurrent neural network module to maintain and update historical information to help the model better understand the current navigation state. The specific calculation formula is as follows:

[0049]

[0050] Among them, LSTM is a classic long short-term memory network recurrent neural network structure, and the input is the visual fusion feature of the current moment and the memory characteristics of the previous moment , the agent updates the memory feature after each predicted action .

[0051] Step S2: construct a natural language instruction key semantic query encoding module to condense and compress the natural language instruction;

[0052] To support more complex and longer natural language instructions and reduce the pressure long sequence inputs place on network model computing resources, the present invention proposes using a Q-Former architecture to condense and compress long sequences of natural language instructions. Condensation refers to summarizing key information, while compression refers to reducing the length of information.

[0053] Specifically, the input of the Q-Former structure is a language instruction. First, a pre-trained text encoder (such as BERT or CLIP) is used to extract the character-level features of the complete natural language instruction. ,in Represents the length of the text. Based on this, a learnable query vector of custom length is randomly initialized ,in Represents the query vector length. This query vector, serving as the query key, interacts with and updates the features of natural language instructions through a multi-head, multi-layer, cross-modal attention module. This process effectively compresses the information of long sequences of instructions into representative key semantics, thereby improving the processing efficiency of the model. Through this mechanism, the network can focus on the key information in natural language instructions, avoiding the waste of computing resources due to redundant information, while improving the accuracy of instruction understanding. The core computational process of this module can be expressed by the following formula:

[0054]

[0055] in, Indicated by A natural language instruction consisting of characters, Indicates the dimension One-hot encoding of represents the position code, represents the language embedding features obtained by one-hot encoding, represents a multi-layer multi-head self-attention pre-trained encoder (such as BERT or CLIP, etc.), represents a multi-layer multi-head cross-modal attention encoder, where the natural language instruction feature encoding are key vectors and value vectors, is the updated text feature.

[0056] Step S3: construct a multimodal conditional cross-attention encoding module, integrate vision, language, local historical displacement and global displacement information, obtain a conditional vector, and provide it to step S4;

[0057] Among them, the multimodal conditional cross attention encoding module consists of two independent multi-layer multi-head cross-modal attention components, which respectively and text features Serving as the query vector of the other modality, realizing the cross-fusion of visual and language modalities, and obtaining the cross-fused visual features respectively With text features , which is calculated as follows:

[0058]

[0059] In order to improve the robot's ability to implicitly locate itself, local historical displacement information is used Indicates the robot's distance from the current time Relative displacement at frame time , respectively represent the time The relative horizontal coordinate, vertical coordinate and heading angle offset of Indicates the displacement information of the current moment relative to the starting point These two types of displacement information are encoded by independent multi-layer perceptrons to obtain displacement features. and , and then take the first token feature of the length of the cross-fused visual features and language features to represent the global features of vision and text respectively (represented as and ), concatenate the above features along the feature dimension to obtain the complete conditional vector: .

[0060] Step S4: construct a conditional diffusion Transformer decoding module, and output multiple candidate multi-step navigation trajectories by learning the denoising process of the multi-step navigation trajectory noise under conditional constraints;

[0061] The diffusion model is a generative model that transforms data into random noise by gradually adding noise (noising), and then gradually removes the noise through the inverse process to restore the original data. In navigation tasks, the diffusion model can generate multi-step navigation trajectories with noise characteristics that meet specific conditions, thus navigating complex and dynamic navigation environments. The core steps involved are as follows:

[0062] Diffusion noise adding process: In the noise adding process, assuming that the original navigation trajectory is (including multi-step information), the diffusion process will add noise to the trajectory, Step iteration to obtain the noisy trajectory The calculation formula for this process is as follows:

[0063]

[0064] in, is the noise coefficient of each step, which is used to control the intensity of noise addition; is the Gaussian noise added at step t.

[0065] Sampling process: Sampling from noise through inverse diffusion process Restore the original navigation track In this process, the model is constructed by conditional features. To guide each step of the denoising operation, thereby generating a navigation trajectory that meets the target conditions. The calculation formula of this process is:

[0066]

[0067] in, is the cumulative noise coefficient of the first t steps, is the conditional feature vector (the conditional vector obtained in step S3), is a noise estimate of the network's predictions.

[0068] Specifically, the noise estimate of the network prediction It is implemented by the Transformer architecture, and the structure is as follows Figure 2 As shown, the conditional diffusion Transformer module consists of The decoder structure of the layer Transformer is stacked, the noise sequence is used as the query input, and the conditional feature vector is used as the key vector and value vector to participate in the calculation of multi-head cross-attention. After the layer decoder iteration, it goes through layer normalization (LayerNorm) and the multi-layer perceptron prediction head to output the predicted noise value.

[0069] Based on a set of random noises, the model can generate a set of candidate multi-step navigation prediction results through the above-mentioned noise addition and denoising processes.

[0070] Step S5: construct a navigation trajectory discriminator based on the contrastive learning method to autonomously determine the matching score between the candidate trajectory and the target point; (e.g. Figure 3 )

[0071] In order to select the most suitable navigation trajectory from multiple groups of candidate navigation trajectories, the present invention constructs a navigation trajectory discriminator based on the contrastive learning method. The input of the discriminator includes the current visual-language fusion feature and the features of each candidate navigation trajectory.

[0072] First, the visual-language fusion feature is extracted through an independent encoder, which uses a self-attention module to process multimodal splicing information from image global features, text global features and historical memory. , and take the average value of token features as the representation, the calculation formula is as follows:

[0073]

[0074] The navigation trajectory features are encoded by another independent multi-layer perceptron encoder to extract the spatial structure and motion pattern of the trajectory:

[0075]

[0076] After feature encoding is completed, the outputs of the two encoders will be compared and learned, and the matching score between each candidate trajectory and the target point is evaluated by calculating the similarity between the visual-linguistic features and the navigation trajectory features. The matching score is calculated as ,in, Indicates the The visual-linguistic input fusion features of input, Indicates the The input trajectory features, Represents the dot product similarity between the visual language input and output trajectories.

[0077] The contrast loss function formula is as follows:

[0078]

[0079] Where, represents the batch size (i.e. the number of input and output pairs), Represents the temperature vector, which is used to adjust the smoothness of the softmax. The larger the temperature vector, the smoother the distribution, and vice versa. This discriminator can efficiently select the navigation trajectory that best matches the current environment and task, thereby improving the accuracy and efficiency of navigation decisions.

[0080] Step S6: Select the navigation trajectory with the highest matching score, and use the MPC trajectory tracking algorithm to control the speed of the mobile robot, completing the interaction with the environment and observation update;

[0081] like Figure 3 As shown, the diffusion model outputs the future The relative offset distance and angle of each step, when applied to robot control, uses the MPC trajectory tracking algorithm to optimize the error between the target trajectory and the actual driving trajectory, and uses the model predictive control framework to adjust the robot's linear speed and angular velocity in real time to ensure that the robot runs smoothly along the selected optimal navigation trajectory, and observes and updates according to environmental changes during the movement, thereby achieving efficient path tracking and dynamic adaptation.

[0082] Specifically, assuming that the future is obtained based on the output of the conditional diffusion model Step reference trajectory , let the discrete time step be , is the linear velocity, is the angular velocity. At the tth moment, the coordinates of the robot are , the heading angle is , then the discrete kinematic model of the robot can be expressed as:

[0083] coordinate:

[0084]

[0085] Heading angle:

[0086]

[0087] in, Representing the The robot coordinates and heading angle at the moment.

[0088] When given the current state and control inputs , MPC calculates the optimal control input by optimizing the cost matrix in the next T time steps. The optimization problem is expressed as:

[0089]

[0090] in, The system is The state of the moment, is the desired state (reference trajectory), is the control input, and is the weight matrix, the trade-off between control state and control input. is a The positive definite matrix is defined as follows:

[0091]

[0092] in, and They represent the weights of the position error in the x and y directions respectively. The larger the value, the more the system values the spatial accuracy of the trajectory. Indicates the weight of the angle deviation. The larger the value, the more the system values the orientation alignment.

[0093] is a positive definite matrix defined as follows:

[0094]

[0095] in, Indicates the penalty weight for linear velocity changes. The larger the value, the more conservative the system will tend to move. Indicates the penalty weight for angular velocity changes. The larger the value, the more likely the system will avoid sharp turns.

[0096] In actual control, MPC only applies the first control input , and re-optimize the control input at the next time step.

[0097] Step S7: Repeat steps S1 to S6 until the stop condition is met or the maximum number of navigation steps is reached, and then the navigation ends.

[0098] Specifically, the stop action is output when one of the following two conditions is met:

[0099] Case 1: The output of the auxiliary stop prediction network branch is greater than the predicted stop threshold. ;

[0100] Case 2: The absolute values of the actions output by the conditional diffusion model are all less than the action stop threshold .

[0101] The auxiliary stop prediction network branch receives the visual global features obtained in step S3 , global features of text and memory characteristics Splicing features As input, it passes through an independent layered perceptron and Sigmoid function to output the stopping probability at the current moment , the distribution range is :

[0102]

[0103] The auxiliary stop prediction network uses mean square error loss for supervision and optimization during training. represents the true value label, and the loss function is expressed as follows:

[0104]

[0105] The selection of the stopping threshold should be based on typical cases during testing. For case 1, the typical value of the predicted stopping threshold can be taken as , for case 2, the typical value of the threshold for stopping the action can be taken as .

[0106] When the stop condition is met, it is considered that the instruction target has been completed and the navigation ends.

[0107] The method of the present invention can effectively improve the accuracy and robustness of the navigation action output of the intelligent agent based on monocular RGB-D input and language instructions. This embodiment simulates a typical scenario where the executed natural language instruction is "Leave the current room from the front door, then turn left and walk into the living room. Pass the dining table and continue to move towards the TV. Stop next to the coffee table and sofa." The output trajectory example is as follows Figure 4 As shown, the first and second rows are the RGB images and depth images obtained by the robot's first-person observation, respectively, and the third row is the multi-step trajectory predicted by the diffusion model. Figure 4 As can be seen in the figure, in the initial state, the model outputs a continuous forward signal, executing the instruction "leave the current room from the front door"; when executing to step 64, the model outputs a continuous left signal, corresponding to the instruction "then turn left"; when executing to step 120, the relative offset of the multi-step trajectory output by the model is very small, corresponding to the stop signal. Based on language instructions, this method can effectively output a local multi-step trajectory for the current situation based on the changing RGB observations during the navigation process, and ultimately successfully reach the correct destination position. Unlike traditional methods that can only output a single-step action (one of left, forward, right, and stop), this method improves flexibility and continuity.

[0108] On the VLN-CE dataset (Vision-and-Language Navigation in Continuous Environments), the proposed method is compared with the classic methods Seq2Seq and CMA. The results are shown in Table 1.

[0109] Table 1 Comparison of experimental results on the VLN-CE dataset

[0110]

[0111] TL and NE represent the navigation length and error length, respectively, in meters; smaller values are better. nDTW, OS, SR, and SPL measure trajectory fit, arrival success rate, stop success rate, and path-weighted stop success rate, respectively, all expressed as percentages; larger values are better. As can be seen, the proposed method significantly improves on all of these metrics compared to the comparison method.

[0112] The above description is only a description of the preferred embodiments of the present application and does not limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical content should be regarded as equivalent valid embodiments and fall within the scope of protection of the technical solution of the present application.

Claims

1. A mobile robot visual language navigation method based on diffusion strategy, characterized in that: The following steps are involved: Step S1: Construct an RGB and depth spatiotemporal fusion encoding module to achieve alignment and fusion of RGB and depth features; Step S2: Construct a natural language instruction key semantic query encoding module to condense and compress the natural language instruction; Step S3: Construct a multimodal conditional cross-attention encoding module to fuse vision, language, local historical displacement and global displacement information to obtain a conditional vector, which is provided to step S4; Step S4: Construct a conditional diffusion Transformer decoding module, obtain a noise vector based on the conditional vector decoding, and act on the multi-step denoising process to output multiple candidate multi-step navigation trajectories; Step S5: construct a navigation trajectory discriminator based on the contrastive learning method to autonomously determine the matching score between the candidate trajectory and the target point; Step S6: Select the navigation trajectory with the highest matching score, combine it with the MPC trajectory tracking algorithm to realize the speed control of the mobile robot, and complete the interaction with the environment and observation update; Step S7: Repeat steps S1 to S6 until the stop condition is met or the maximum number of navigation steps is reached, and navigation ends.

2. The mobile robot visual language navigation method based on diffusion strategy according to claim 1 is characterized in that: In step S1, the input of the constructed RGB and deep spatiotemporal fusion encoding module is the RGB color map and depth map returned by the current robot camera. The features are extracted by the pre-trained model respectively, and the alignment and fusion of RGB and depth features are achieved through weighted learning. Then, LSTM is used to update and integrate the temporal information to obtain Moment Memory Features .

3. The mobile robot visual language navigation method based on diffusion strategy according to claim 1 is characterized in that: In step S2, the input of the constructed natural language instruction key semantic query encoding module is the complete input string. The pre-trained text encoder is used to extract features of different character positions. In order to reduce the computational burden and attention dispersion problem caused by long texts, the Q-Former structure is used to further extract key semantic information. The calculation formula is as follows: in, Indicated by A natural language instruction consisting of characters, Indicates the dimension One-hot encoding of represents the feature dimension, represents the position code, represents the language embedding features obtained by one-hot encoding, represents a multi-layer multi-head self-attention pre-trained encoder, represents the learnable query variable, is the query vector length, represents a multi-layer multi-head cross-modal attention encoder, where the natural language instruction feature encoding are key vectors and value vectors, is the updated text feature.

4. The mobile robot visual language navigation method based on diffusion strategy according to claim 1 is characterized in that: In step S3, the multimodal conditional cross attention encoding module is composed of two independent multi-layer multi-head cross-modal attentions, which respectively make the visual features and text features Serving as the query vector of the other modality, realizing the cross-fusion of visual and language modalities, and obtaining the cross-fused visual features respectively With text features ; Local historical displacement information The robot is the distance from the current time Relative displacement at frame time , respectively represent the time Relative horizontal coordinate, vertical coordinate and heading angle offset; global displacement information is the displacement information relative to the starting point at the current moment ; These two types of displacement information are encoded by independent multi-layer perceptrons to obtain displacement features and ,in ; Then take the first token feature of the length after cross-fusion of visual features and language features, representing the visual global features and global features of text , concatenate the above features along the feature dimension to obtain the complete conditional vector.

5. The mobile robot visual language navigation method based on diffusion strategy according to claim 4 is characterized in that: In step S4, the query input of the conditional diffusion Transformer structure is a set of randomly sampled Gaussian noise ,in Represents the length of the output trajectory, and 3 represents the output of the relative displacement prediction at each moment , the conditional feature is the conditional vector obtained in step S3, combined with the sampling steps , training and sampling are completed based on the diffusion model principle. Due to the random characteristics of the diffusion model, a set of candidate multi-step navigation trajectory prediction results will be obtained.

6. The mobile robot visual language navigation method based on diffusion strategy according to claim 5 is characterized in that: The diffusion model is a generative model that converts data into random noise through a step-by-step noise addition process, and then gradually denoises the data through an inverse process to restore the original data. The specific steps are as follows: Diffusion noise adding process: In the noise adding process, assuming that the original navigation trajectory is , including multi-step Information, the diffusion process will add noise to the trajectory, after Step iteration to obtain the noisy trajectory ; The calculation formula of this process is as follows: in, is the noise coefficient at each step, which controls the intensity of noise addition; is the Gaussian noise added at step t; Sampling process: Sampling from noise through inverse diffusion process Restore the original navigation track ; In this process, the model is To guide the denoising operation at each step, thereby generating a navigation trajectory that meets the target conditions; the calculation formula of this process is: in, is the cumulative noise coefficient of the first t steps, is the conditional eigenvector, is the noise estimate of the network prediction; Noise estimation of network predictions It is implemented by the Transformer architecture, where the conditional diffusion Transformer module is composed of The decoder structure of the layer Transformer is stacked, the noise sequence is used as the query input, and the conditional feature vector is used as the key vector and value vector to participate in the calculation of multi-head cross-attention; After the layer decoder is iterated, it passes through the layer normalization LayerNorm and the multi-layer perceptron prediction head to output the predicted noise value.

7. The mobile robot visual language navigation method based on diffusion strategy according to claim 1 is characterized in that: In step S5, a navigation trajectory discriminator is constructed based on the contrastive learning method. The input of the discriminator is the current visual-language fusion feature and the navigation trajectory feature. Feature encoding is completed by two independent encoders respectively. The discriminator is trained using the contrastive loss function, and its formula is as follows: Where, Indicates the batch size, Indicates the The visual-linguistic input fusion features of input, Indicates the The input trajectory features, Indicates the The visual language input and The dot product similarity of the predicted trajectories, Indicates the The visual language input and The dot product similarity of the predicted trajectories, Indicates the The predicted trajectory and The dot product similarity of the visual language input, Represents the temperature vector, which is used to adjust the smoothness of the soft maximization.

8. The mobile robot visual language navigation method based on diffusion strategy according to claim 1 is characterized in that: The MPC trajectory tracking algorithm in step S6 optimizes the error between the target trajectory and the actual driving trajectory, and uses the model predictive control framework to perform real-time speed adjustment to ensure that the robot runs smoothly along the selected optimal navigation trajectory. It also observes and updates according to environmental changes during the movement process, thereby achieving efficient path tracking and dynamic adaptation.

9. The mobile robot visual language navigation method based on diffusion strategy according to claim 1 is characterized in that: In step S7, a stop action is output when one of the following two conditions is met: Case 1: The output of the standalone auxiliary stop prediction network branch has a stop probability greater than the predicted stop threshold. ; Case 2: The absolute values of the actions output by the conditional diffusion model are all less than the action stop threshold ; The auxiliary stop prediction network branch receives the visual global features obtained in step S3 , global features of text and memory characteristics Splicing features As input, , after an independent layered perceptron and Sigmoid function, output the stopping probability at the current moment , the distribution range is : The auxiliary stop prediction network is supervised and optimized using mean square error loss during training. represents the true value label, and the loss function is expressed as follows: 。

Images