A motion adaptive method and system for robot vision and language navigation

Abstract

The present invention relates to an action adaptation method and system for robot visual language navigation, including: determining the numerical range of the robot's movable action space and generating random actions; based on the changes obtained by the interaction between the random actions and the environment, using the robot action to affect the self-encoding module to obtain the action embedding vector; based on the action embedding vector, comparing the specific actions in the data set and calculating the difference, determining the current execution action and the corresponding semantic meaning; based on the deterministic action and the interaction with the environment, using the cross-modal attention feature extraction module to obtain the robot visual language multimodal features; based on the robot visual language multimodal features, as the diffusion strategy decision module conditions, iteratively inferring the robot's next execution action; updating the robot state until the target is reached and the navigation task is terminated. Compared with the prior art, the present invention has the advantages of enhancing the robustness and accuracy of the robot's action adaptation.

Description

Technical Field

[0001] The present invention relates to the technical field of robot vision language navigation, and in particular to a motion self-adaptation method and system for robot vision language navigation. Background Art

[0002] Robots can demonstrate intelligence similar to that of humans or living things through autonomous or semi-autonomous perception, movement, operation or human-computer interaction; they can extend the constraints and restrictions on humans in terms of scale, time, space, environment, emotion, intelligence, precision, speed, power, etc., and serve humans. Robots that can communicate with humans using natural language, understand the surrounding environment, and complete human instructions have long been a goal of mankind. The robot multimodal navigation task based on vision and language is a research topic proposed to achieve this goal, and has received widespread attention in recent years. The robot vision and language navigation task refers to letting the robot navigate in accordance with natural language instructions, while understanding the language instructions and the image information that can be seen in the robot's first-person perspective, realizing the fusion and matching of multimodal information, and then adjusting and repairing its own state in the environment, predicting navigation actions, such as turning and moving forward, and executing corresponding actions to finally reach the target location. For example, we put the robot in a bedroom and let it pick up a notebook in another bedroom. Because there is no line of sight in the room, the robot needs to walk out of the room, pass through corridors, living rooms and other places, and decide the next action by understanding and reasoning about the environmental information in each frame of the picture observed, and finally locate and find the notebook.

[0003] In recent years, the research field of embodied intelligence has been very hot. In layman's terms, embodied intelligence means that machines have the ability to understand, reason and interact with the physical world. Robot vision and language navigation tasks are a relatively popular and representative research topic, and are also an important step for intelligent robots to enter human life. Simply put, the robot vision and language navigation task is a task for robots to observe the environment, perform actions, interact with the environment, and find the target according to given language instructions. By learning autonomous navigation, robots can get rid of their dependence on environmental maps, be deployed to various tasks more quickly, and facilitate task expansion. However, all these ideas are based on the assumption that robots can perfectly execute several standard discrete navigation action instructions by default in tasks. In actual scenarios, robots may face a series of challenges, including manufacturing accuracy errors, noise interference caused by sensor failures such as motors, and uncertainty disturbances caused by malicious attacks, as well as migration and generalization between different robots. Therefore, in the robot vision and language navigation task, we should not only focus on solving input problems, such as improving the matching degree of multimodal information fusion, but also consider that robot learning methods also need to focus on output problems, such as motion constraints that actual robots have to face, so as to better promote the development of embodied intelligence research.

[0004] The problem of motion adaptation in robot vision and language navigation is that when errors occur in robot modeling, or when there is a fault in the robot actuator, or when the robot type is changed, the robot's action in performing the navigation task is no longer the action in the previous training stage. That is, in the test task, the robot's action space is different from the action space in the training task. At present, the problem of motion adaptation in robot vision and language navigation is still in a state of no concern. The main reason is that it is difficult to solve the generalization of discrete action space in reinforcement learning. Generally, either continuous action space training is carried out directly, or the parameters of the robot in the test phase are identified and adjusted to the discrete action space configuration of the robot in the training phase. However, training continuous action space for robot vision and language navigation tasks is very time-consuming and resource-consuming. Because of this limitation, the configuration of the robot's action space in the current field of large-scale robot model research, which is in an unprecedented state, is also based on discrete action space. And parameter identification and adjustment of discrete action space parameters for possible situations such as errors in robot modeling, faults in robot actuators, or changes in robot type is also a very time-consuming and large project.

[0005] The diffusion model is a generative model that aims to learn how to generate similar data given some training data. The essence of the generative model is to fit the given data sample through a known probability model. The diffusion model is implemented by training a parameterized Markov chain through variational inference. In specific implementation, the generation of the diffusion model starts with random noise. The original image in the input data set is first gradually added to the data through the forward process, and then the noise added at each step is predicted through the reverse process. The noise is gradually restored by removing the noise. After iterative refinement in the diffusion process, the noise-free generated image with the same distribution as the original image is finally output. The core idea is that in each step, the model estimates how to generate a denoised version from the current input. After such a cyclic iteration, a pure noise image randomly sampled from a standard Gaussian distribution can be sampled through step-by-step denoising to generate a real image with the same distribution as the original image.

[0006] After searching, Chinese invention patent publication number CN115082915B discloses a mobile robot visual-language navigation method based on multimodal features, including: 1) encoding the input natural language instructions to obtain a language feature vector and an initial state feature vector; 2) encoding the input image at the current moment to obtain a visual feature vector; 3) encoding the historical action of the robot to obtain a historical feature vector; 4) cross-modal encoding of the feature vectors of multiple modes obtained from step 1) to step 3); 5) inputting the cross-modal encoding results into the action decision module to predict the next action and execute it; 6) repeating steps 2) to 5) until the robot stops moving, and then updating the model. The present invention provides object reference information and historical navigation information, which enables the robot to perform visual navigation according to natural language instructions. Experiments show that the proposed method has excellent performance. The existing patent has the problem that when the robot performs unknown changes in action, it cannot be robust to new actions that have not been seen.

[0007] How to achieve robot motion adaptation under strong constraints in robot vision and language navigation tasks has become a technical problem that needs to be solved. It is also the only way to achieve the generalization of heterogeneous robot motions in vision and language navigation tasks and move towards a general navigation robot. Summary of the invention

[0008] The purpose of the present invention is to provide a method and system for motion adaptation of robot vision language navigation in order to overcome the defect that the robot cannot be robust to new and unseen motions when unknown changes occur in the robot's motion execution under strong constraints in robot vision language navigation tasks.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] According to one aspect of the present invention, a method for motion adaptation of robot visual language navigation is provided, wherein the method is used for a visual language navigation task with uncertain robot motion, and comprises the following steps:

[0011] Step S101, determining the numerical range of the robot's movable action space and generating random actions;

[0012] Step S102, based on the changes obtained by the interaction between the random action and the environment, the robot action is used to influence the self-encoding module to obtain an action embedding vector;

[0013] Step S103, based on the action embedding vector, compare the specific actions in the data set and calculate the difference to determine the action currently executed by the robot and the semantic meaning corresponding to the action;

[0014] Step S104, based on the interaction between the current execution action and the environment, a cross-modal attention feature extraction module is used to obtain multimodal features of the robot visual language;

[0015] Step S105, using the robot's visual language multimodal features as the diffusion strategy decision module conditions, iteratively inferring the robot's next execution action, and updating the robot's state information;

[0016] Step S106, repeat steps S102 to S105 until the robot reaches the vicinity of the target, thus ending the navigation task.

[0017] Preferably, the numerical range of the robot's movable action space includes a numerical range of forward step length and a numerical range of turning angle.

[0018] Preferably, the changes obtained based on the interaction between random actions and the environment are specifically: the robot interacts with the visual language navigation environment by performing random actions, and obtains first-person perspective observation pictures before and after the robot performs the current action.

[0019] More preferably, the obtaining of the action embedding vector by influencing the self-encoding module using the robot action comprises:

[0020] First, observe the first-person perspective images o of the robot before and after performing the current action. t-1 and t , perform pre-trained ResNet50 network visual encoding and linear transformation dimension processing to obtain the extracted visual feature r t ;

[0021] Then, the robot currently performs action a t-1 =a i Index to obtain the memory vector matrix corresponding to a i The associated latest memory vector mi ;

[0022] Finally, the memory vector m i and visual features r t Passed to the recurrent neural network, the update is done by action a i The action embedding vector e that causes the state change i .

[0023] More preferably, the determining the action currently executed by the robot and the semantic meaning corresponding to the action includes:

[0024] First, the relevant actions in the specific discrete action space in the data set are indexed to obtain the corresponding action embedding vectors in the action embedding vector matrix;

[0025] Then, the difference between the action embedding vector of the robot's current action and the action embedding vector of the corresponding action in the specific discrete action space in the data set is calculated by cosine similarity, and the semantic meaning of the current action is obtained by comparison, that is, the specific value of the forward step length or turning angle of the robot's current action;

[0026] Finally, these values ​​are used as fixed measurement values ​​for the relevant actions in the discrete action space of the robot in this visual language navigation task.

[0027] Preferably, the method of using a cross-modal attention feature extraction module to obtain robot visual language multimodal features includes:

[0028] First, based on the deterministic actions, the robot interacts with the environment to obtain the first-person perspective image o t and natural language navigation trajectory instructions;

[0029] Then, for the RGB frame ov t Observation values, using pre-trained ResNet50 to collect semantic visual spatial features V t = {v i}, where i represents the spatial position index, and then average pooling is performed; for the depth frame od t observations, using a modified ResNet50 trained to perform point target navigation and represent these features D t =[d1,…,d wh ], where w represents the image width and h represents the image height; the natural language navigation trajectory instructions are converted into corresponding GLoVE embeddings, and features are extracted using a bidirectional long short-term memory network;

[0030] Finally, a Transformer-based cross-modal attention feature extraction model is used to extract visual-linguistic multimodal features between visual images and language instructions, where the visual-linguistic multimodal features include: natural language instruction deep features Visual RGB-Natural Language Command Features Visual Depth-Natural Language Command Features And the hidden layer information h of the GRU network t , specifically:

[0031]

[0032]

[0033]

[0034] Among them, Attn represents the scaled dot product attention mechanism, Represents the natural language instruction feature S and the hidden layer information h of the GRU network t The deep features of natural language instructions extracted after the cross-attention mechanism, Represents the visual RGB feature V t Deep features with natural language instructions The visual RGB-natural language instruction features extracted after the cross-attention mechanism, Represents the visual depth feature D t Deep features with natural language instructions Visual depth-natural language command features extracted after cross-attention mechanism.

[0035] Preferably, the robot visual language multimodal features are used as the diffusion strategy decision module conditions to iteratively infer the next action of the robot. Specifically, the robot visual language multimodal features are used as the diffusion strategy decision module conditions to generate the conditional probability distribution p(a t |c t ), the diffusion strategy starts from the sampled noise action A t (k) Iterate denoising step by step until a noise-free action A is restored iteratively t (0 is used as the next execution action of the robot, and the iterative denoising process is as follows:

[0036] x k-1 =α(x k -γε θ (x k ,k)+N(0,σ 2 I))

[0037] Among them, α, γ, σ represent the correlated noise learning rate, ε θ represents the noise prediction network, N(0,σ 2 I) represents the Gaussian noise added at each iteration, x k Represents the k-th iteration denoising action.

[0038] Preferably, the robot state information includes a first-person-view observation picture of the robot and position information, wherein the first-person-view observation picture includes an RGB frame and a depth frame.

[0039] According to one aspect of the present invention, there is provided an action adaptation system for robot visual language navigation, the system comprising a feasible space numerical range determination module, an action impact self-encoding module, an action metric value solving module, a cross-modal attention feature extraction module and a diffusion strategy decision module;

[0040] The feasible space value range determination module is used to determine the range of measurement values ​​of the robot's executable actions in the visual language navigation task and provide a basis for random value selection;

[0041] The action impact self-encoding module is used to obtain the impact result of the robot's current action based on the interaction between random actions and the environment, and obtain the action embedding vector of the robot's current action in the visual language navigation task;

[0042] The action metric value solving module is used to compare the action embedding vector of the currently executed action with the corresponding action in the specific discrete action space in the data set, calculate the difference between the two vectors by cosine similarity, and obtain the semantic meaning corresponding to the currently executed action, so as to determine the metric value of the action in the new discrete action space of the robot in this navigation task;

[0043] The cross-modal attention feature extraction module is used to extract visual language action multimodal features based on the robot's first-person perspective observation pictures and natural language navigation trajectory instructions obtained by interacting with the environment through deterministic actions, so as to obtain the robot's visual language multimodal features;

[0044] The diffusion strategy decision module is used to perform conditional denoising diffusion based on the multimodal features of the robot's visual language to iteratively restore the sampled noisy action into a noise-free action as the robot's next execution action.

[0045] Preferably, the action impact self-encoding module comprises a visual feature extraction unit and an action embedding vector generation unit connected in sequence;

[0046] The visual feature extraction unit is used to obtain the first-person perspective observation pictures of the robot before and after performing the current action based on the interaction between the random action and the environment. t-1and t , perform pre-trained ResNet50 network visual encoding and linear transformation dimension processing to obtain the extracted visual feature r t ;

[0047] The action embedding vector generating unit is used to generate an action embedding vector based on the current action a performed by the robot. t-1 =a i Index to obtain the memory vector matrix corresponding to a i The latest memory associated with m i , the memory vector m i and visual features r t Passed to the recurrent neural network, the update is done by action a i The action embedding vector e that causes the state change i .

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1) Based on the changes obtained by the interaction between random actions and the environment, the present invention uses the action influence self-encoding method that different robot actions have different effects on the environment to obtain the action embedding vector, which can realize robot action fault diagnosis, calculate the measurement value of unknown execution action, accurately identify the robot execution action, and can be generalized to unknown data distribution under the strong constraints of the robot visual language navigation task, and meet the balance between independent and identical distribution of training data sets and generalization outside the distribution in the test stage, thereby enhancing the robustness of robot action adaptation and realizing end-to-end robot autonomous and stable navigation.

[0050] 2) The present invention extracts the multimodal features of robot visual language through the constructed cross-modal attention feature extraction model, and uses the denoising diffusion model based on the multimodality of visual language as the action decision strategy. This strategy can adapt to new discrete action spaces that have not been seen before, and can improve the accuracy of actions generated by the action diffusion strategy, and has good robustness. Moreover, this method is independent of the feature extraction model and can be combined with any other visual language feature extraction model. It can be applied to heterogeneous robot visual language navigation tasks, and is of great significance to the development of general navigation robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic flow chart of a method for adaptive motion of robot visual language navigation provided by an embodiment of the present invention;

[0052] Figure 2 A schematic flow chart of another method for motion adaptation of robot visual language navigation provided by an embodiment of the present invention;

[0053] Figure 3A schematic diagram of a motion adaptive system for robot visual language navigation provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of another motion adaptive system for robot visual language navigation provided by an embodiment of the present invention;

[0055] In the accompanying drawings, 10 is a feasible space numerical range determination module, 20 is an action impact self-encoding module, 30 is an action metric value solution module, 40 is a cross-modal attention feature extraction module, 50 is a diffusion strategy decision module, 21 is a visual feature extraction unit, and 22 is an action embedding vector generation unit. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0057] This embodiment relates to a motion adaptive method for robot visual language navigation, which is applied to robot visual language navigation tasks with uncertain motion, such as Figure 1 , the method specifically comprises the following steps:

[0058] Step S101, determine the numerical range of the robot's movable action space and generate random actions.

[0059] Specifically, in the visual language navigation task, the robot uses low-level actions to perform tasks in a continuous environment, mainly by moving forward a fixed step distance and turning a fixed angle to achieve movement. For example, the configuration of the specific discrete action space in the dataset is {move forward 0.25m, turn left 15°, turn right 15°, stop}, and its corresponding Chinese is {move forward 0.25m, turn left 15°, turn right 15°, stop}, that is, the fixed step length is 0.25 meters and the fixed angle is 15 degrees. The action "stop" means that the robot has reached the vicinity of the target and completed this navigation task.

[0060] In the embodiment of the present invention, the action measurement value of the robot in the visual language navigation task is unknown at the beginning, which is a problem of action uncertainty, but the prerequisite is that the unknown action is an action that the robot can perform, so it is necessary to determine the numerical range of the robot's movable action space, that is, to determine the numerical range of the forward step length and turning angle of the robot's action, to provide a basis for random value taking or manual selection, and the values ​​within the range are likely to become the measurement values ​​of the robot's initial random action.

[0061] In the embodiment of the present invention, after multiple experiments, the robot's forward step length range is set to [0.10m, 2.0m], the turning angle range is set to [1°, 30°], and the minimum spacing is set to 0.01m and 1°.

[0062] Step S102, based on the changes obtained by the interaction between the random action and the environment, the robot action is used to influence the self-encoding module to obtain an action embedding vector.

[0063] In the embodiment of the present invention, the robot relies on the impact of the action to reflect its predefined semantic meaning.

[0064] Specifically, the robot interacts with the visual language navigation environment by performing random actions, and obtains the first-person observation pictures before and after the robot performs the current action. t-1 and t , where the current time step robot's first-person perspective observation picture includes the RGB frame ov t and depth frame od t The role of the robot action impact autoencoder module is to update the action embedding vector matrix based on the changes in the environment caused by the actions taken by the robot in the visual language navigation task, that is, through adjacent observations.

[0065] First, observe the first-person perspective images o of the robot before and after performing the current action. t-1 and t Perform pre-trained ResNet50 network visual encoding and linear transformation dimension processing to obtain the extracted visual feature r t .

[0066] Then, the robot currently performs action a t-1 =a i Index to obtain the memory vector matrix corresponding to a i The associated latest memory vector m i .

[0067] Finally, the memory vector m i and visual features r t The loop is passed to the recurrent neural network, and the loop update is performed by action a i The action embedding vector e that causes the state change i Because only the forward step length and turning angle will change uncertainly in the robot's actions, and the stop action means that the robot has completed and will end this task, that is, it has deterministic action semantics, so a special action embedding vector e is directly specified stop To indicate that there is no need to go through the action-affecting self-encoding module cyclic update.

[0068] Step S103, based on the action embedding vector, compare the specific actions in the data set to calculate the difference and determine the semantic meaning corresponding to the current action.

[0069] In an embodiment of the present invention, each navigation trajectory of the data set consists of a natural language navigation trajectory instruction and a state-action pair, wherein the state-action pair is a specific action in a specific discrete action space in the data set corresponding to each robot first-person perspective observation image, and each action has specific semantics.

[0070] Specifically, the three actions of moving forward 0.25m, turning left 15° and turning right 15° in the specific discrete action space in the dataset are indexed, and the corresponding action embedding vectors in the action embedding vector matrix are obtained; the action embedding vector of the currently executed action is compared with the action embedding vector of the corresponding action in the specific discrete action space in the dataset, and the difference between the two vectors is calculated by cosine similarity to obtain the semantic meaning of the currently executed action, that is, the specific value of the forward step length or turning angle of the robot's current action, and then the new discrete action space of the current robot is updated, and these values ​​are used as fixed measurement values ​​for related actions in the discrete action space of the robot in this visual language navigation task.

[0071] Step S104, based on the interaction between the deterministic action and the environment, a cross-modal attention feature extraction module is used to obtain multimodal features of the robot visual language.

[0072] Specifically, the robot interacts with the visual language navigation environment by performing deterministic actions to obtain the first-person observation image o of the robot at the current time step. t and natural language navigation trajectory instructions, where the RGB frame ov t and depth frame od t The observation values ​​are used for image classification and point target navigation respectively, and the natural language navigation trajectory instructions are used to constrain the robot's navigation path to prevent the robot from over-exploring the environment.

[0073] First, the RGB frame ov t Observations, using ResNet50 pre-trained on ImageNet to collect semantic visual spatial features V t = {v i}, where i represents the spatial position index, and then average pooling (mean-pool) is performed; for the depth frame od t observations, using a modified ResNet50 trained to perform point target navigation (i.e., navigating to a location given in relative coordinates) and represent these features D t =[d1,…,d wh], where w represents the image width, h represents the image height, and [] represents concatenation; for natural language navigation trajectory instructions (such as "Leave the bedroom, and enter the kitchen. Walk forward, and take a left at the couch. Stop in front of the window.", which corresponds to the Chinese "Leave the bedroom, enter the kitchen. Walk forward, and take a left at the couch. Stop in front of the window."), convert it into the corresponding GLoVE embedding (global vector word embedding), encoded as (w1,…,w T ), where T represents the instruction length, and the feature S is extracted using a bidirectional long short-term memory network. These processes are expressed as:

[0074]

[0075]

[0076] S=BiLSTM(w1,…,w T ) (3)

[0077] Then, because the robot visual language navigation task is a multimodal task, a simple sequence-to-sequence model cannot achieve cross-modal fusion and spatial visual reasoning. Considering the long-distance dependency modeling capability of the Transformer model, it can be used to extract deeper multimodal information between visual images and language instructions, so a cross-modal attention feature extraction model is constructed. The decisive role in the Transformer model is the attention mechanism, which can focus on the connection within its own modal information, as well as the connection between cross-modal information. Therefore, the cross-modal attention feature extraction process is expressed as:

[0078]

[0079] Among them, GRU is a model structure in the recurrent neural network. t-1 represents the linear vector obtained through learning of the robot’s current action, h t-1 and h t Represents the hidden layer information of the GRU network, Represents visual depth features.

[0080]

[0081] Among them, Attn represents (scaled dot-product attention) scaling dot product attention mechanism, That is, it represents the natural language instruction feature S and the hidden layer information h of the GRU networkt The deep features of natural language instructions extracted after the cross-attention mechanism, That is, the visual RGB feature V t Deep features with natural language instructions The visual RGB-natural language instruction features extracted after the cross-attention mechanism, That is, the visual depth feature D t Deep features with natural language instructions Visual depth-natural language command features extracted after cross-attention mechanism.

[0082] Finally, After splicing, the multimodal features of robot vision language are obtained.

[0083] Step S105, based on the multimodal features of the robot's visual language, as the conditions of the diffusion strategy decision module, iteratively infer the robot's next execution action.

[0084] In the embodiment of the present invention, the configuration of the discrete action space in the state-action pair of the data set is {moveforward 0.25m, turn left 15°, turn right 15°, stop}, which corresponds to Chinese as {move forward 0.25m, turn left 15°, turn right 15°, stop}, that is, the fixed step length is 0.25 meters, and the fixed angle is 15 degrees. The present invention is dedicated to solving the problem of uncertainty in action measurement values ​​in robot visual language navigation tasks, and needs to be able to adapt to new actions that have not been seen in the data set. Therefore, a new conditional denoising diffusion model based on visual language multimodality is introduced as an action decision strategy, which can adapt to new discrete action spaces that have not been seen.

[0085] Specifically, the diffusion model is a probabilistic generative model that iteratively refines the randomly sampled noise into samples from the underlying distribution. The diffusion model can naturally and accurately express multimodal distributions from two aspects: the basic random sampling process and random initialization. Each sampling process starts by drawing an initial sample A from a standard Gaussian distribution. t (k), which helps predict A for the final action t (0) specifies different possible convergence points, and in a large number of iterations, the action is further stochastically optimized using the added Gaussian perturbation, so that a single sample action converges and moves between different multimodal action distributions.

[0086] First, the diffusion model is a type of generative model in which the output generation is simulated as a denoising process. k Initially, the diffusion model performs k iterations of denoising, generating a series of intermediate actions x with decreasing noise levels.k , x k-1 , …, x0, until the desired noise-free output x0 is formed. This process follows the following formula:

[0087] x k-1 =α(x k -γε λ (x k ,k)+N(0,σ 2 I)) (6)

[0088] Among them, α, γ, σ represent the correlated noise learning rate, ε θ represents the noise prediction network, which is the core of diffusion model learning. 2 I) represents the Gaussian noise added at each iteration.

[0089] Conceptually, it can also be understood as learning a gradient field for the implicit action score and then optimizing this gradient during inference:

[0090]

[0091] Among them, understanding the gradient field Equivalent understanding of noise prediction network ε θ , x represents the initial score of the action, γ represents the gradient field learning rate, and x′ represents the updated action score.

[0092] Then, the first step of the diffusion model training process is to randomly extract samples x0 from the dataset, and for each sample, randomly select a denoising iteration number k and randomly sample a Gaussian noise ε k , we can define the loss function of the diffusion model as:

[0093] LOSS=MSE(ε k ,ε θ (x0+ε k ,k)) (8)

[0094] Among them, ε θ For the noise prediction network.

[0095] In an embodiment of the present invention, although the data set only contains data in a specific discrete action space, it can still be used as a sampling source for the diffusion model training process, because the diffusion model can naturally and accurately express multimodal distributions. After the action measurement value of the robot's new discrete action space is determined, the action diffusion strategy can make correct decisions based on the learned capabilities.

[0096] Finally, based on the robot vision language multimodal feature VL t , as the condition of the robot action diffusion strategy decision model, where the robot visual language multimodal feature VLt Using the Transformer model to associate with the action embedding vector, the action diffusion decision denoising process at the current time step can be expressed as:

[0097] A k-1 =α(A k -γε θ (VL t ,A k ,k)+N(0,σ 2 I)) (9)

[0098] Among them, A k-1 and A k represents the intermediate noise action in the diffusion model denoising process, and A k After one step of denoising, A k-1 ; k is a randomly selected integer used to obtain A from the noiseless action A0 after k times of noise addition k ; VL t represents the multimodal features of robot visual language, α, γ, σ represent the related noise learning rate, ε θ represents the noise prediction network, which is the core of diffusion model learning; N(0,σ 2 I) represents the Gaussian noise added at each iteration.

[0099] The corresponding loss function is:

[0100] LOSS=MSE(ε k ,ε θ (VL t ,A0+ε k ,k)) (10)

[0101] Among them, ε k It means that the noiseless action A0 is obtained by adding noise k times k The equivalent Gaussian noise, MSE is the mean square error (MeanSquared Error), which is used to calculate ε k and ε θ The error LOSS between the noise predicted by the network.

[0102] Using robot visual language multimodality as a constraint can improve the accuracy of action generation by the action diffusion strategy and help make the action adaptation process of robot visual language navigation stable and feasible.

[0103] Example 2

[0104] The present embodiment also relates to a motion adaptation method for robot visual language navigation, which introduces a method for self-encoding the impact of different actions on the environment. The motion embedding vector obtained by modeling can realize robot motion fault diagnosis and calculate the measurement value of the executed action; at the same time, the present invention introduces a new conditional denoising diffusion model based on visual language multimodality as an action decision strategy, which can adapt to new discrete action spaces that have not been seen before; at the same time, the motion adaptation method provided by the embodiment of the present invention can realize end-to-end autonomous and stable navigation, and solves the technical problem that when the robot executes the action under strong constraints in the robot visual language navigation task and undergoes unknown changes, it cannot be robust to new actions that have not been seen before.

[0105] Figure 2 FIG. 1 is a flow chart of another method for motion adaptation of robot visual language navigation provided according to an embodiment of the present invention. Figure 2 As shown, the method specifically comprises the following steps:

[0106] Step S201, initialize the initial state of the robot, randomly or manually select actions, and give natural language navigation trajectory instructions according to the environment.

[0107] Step S202, based on the unknown action, the robot interacts with the environment to obtain first-person perspective observation pictures before and after the robot performs the current action.

[0108] Step S203: Based on the two observation pictures before and after the robot, the action impact self-encoding module is input to obtain the action embedding vector of the robot's current action.

[0109] Step S204, compare the action embedding vector with the corresponding action embedding vector in the data set for similarity, and determine the metric value of the action in the new discrete action space of the robot in this task.

[0110] Step S205, input the robot's first-person perspective observation picture and natural language instructions into the cross-modal attention feature extraction module to obtain the robot's visual language multimodal features.

[0111] Step S206, based on the multimodal features of the robot's visual language, as the conditions of the diffusion strategy decision module, iteratively calculate the robot's next execution action.

[0112] Step S207, interacting with the environment based on the decision action, updating the robot's first-person perspective observation picture and position information.

[0113] Step S208, determine whether the target is reached, if not, execute step S209; if the target is reached, execute step S210.

[0114] Step S209, determine whether there are any unknown actions. If so, jump to step S203 to perform iterative calculations; if all the actions are known, jump to step S205 to perform iterative calculations.

[0115] Step S210: the robot performs a stop action and immediately ends the navigation task.

[0116] Example 3

[0117] This embodiment also relates to a motion adaptive system for robot vision and language navigation, which is applied to robot vision and language navigation tasks with uncertain motions.

[0118] like Figure 3 As shown, the system includes a feasible space numerical range determination module 10, an action impact self-encoding module 20, an action metric value solution module 30, a cross-modal attention feature extraction module 40 and a diffusion strategy decision module 50.

[0119] Specifically, the feasible space numerical range determination module 10 is used to determine the measurement value range of the robot's executable actions in the visual language navigation task, providing a basis for random value selection, and the values ​​within the range are likely to become the measurement values ​​of the initial random action.

[0120] The action impact self-encoding module 20 is used to interact with the environment based on the random action to obtain the impact result of the robot's current action, and obtain the action embedding vector of the robot's current action in the visual language navigation task.

[0121] The action metric value solving module 30 is used to compare the action embedding vector of the current action performed by the robot with the corresponding action in the specific discrete action space in the data set, calculate the difference between the two vectors by cosine similarity, and obtain the semantic meaning corresponding to the current action, so as to determine the metric value of the action in the new discrete action space of the robot in this navigation task.

[0122] The cross-modal attention feature extraction module 40 is used to extract visual language action multimodal features based on the robot's first-person perspective observation pictures and natural language navigation trajectory instructions obtained by interacting with the deterministic action and the environment, so as to obtain the robot's visual language multimodal features.

[0123] The diffusion strategy decision module 50 is used to perform conditional denoising diffusion based on the multimodal features of the robot visual language to iteratively restore the sampled noise action to a noise-free action as the next execution action of the robot.

[0124] Example 4

[0125] The present embodiment also relates to a motion adaptation system for robot visual language navigation, which introduces a method for self-encoding motion effects according to different effects of different actions on the environment. The motion embedding vector obtained through modeling can realize robot motion fault diagnosis and calculate the measurement value of the executed action. At the same time, the present invention introduces a new conditional denoising diffusion model based on visual language multimodality as an action decision strategy, which can adapt to new discrete action spaces that have not been seen before. At the same time, the motion adaptation method provided by the embodiment of the present invention can realize end-to-end autonomous and stable navigation, and solves the technical problem that when the robot executes actions under strong constraints in the robot visual language navigation task and undergoes unknown changes, it cannot be robust to new actions that have not been seen before.

[0126] Optionally, Figure 4 FIG. 1 is a schematic diagram of another motion adaptive system for robot visual language navigation provided according to an embodiment of the present invention. Figure 4 As shown, the model building module 20 includes a visual feature extraction unit 21 and an action embedding vector generation unit 22.

[0127] Specifically, the visual feature extraction unit 21 is used to obtain the first-perspective observation pictures of the robot before and after performing the current action based on the interaction between the random action and the environment. t-1 and t , perform pre-trained ResNet50 network visual encoding and linear transformation dimension processing to obtain the extracted visual feature r t .

[0128] The action embedding vector generating unit 22 is used to generate an action embedding vector based on the action a currently executed by the robot. t-1 =a i , index and obtain the memory vector matrix corresponding to a i The latest memory associated with m i , the memory vector m i and visual features r t The loop is passed to the recurrent neural network and the update is performed by action a i The action embedding vector e that causes the state change i .

[0129] Because only the forward step length and turning angle will change uncertainly in the robot's actions, and the stop action means that the robot has completed and will end this task, that is, it has deterministic action semantics, so a special action embedding vector e is directly specified stop To indicate that there is no need to go through the action-affecting self-encoding module cyclic update.

[0130] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0131] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0132] The processing unit performs the various methods and processes described above, such as methods S101 to S105, S201 to S210. For example, in some embodiments, methods S101 to S105, S201 to S210 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S101 to S105, S201 to S210 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S101 to S105, S201 to S210 in any other appropriate manner (e.g., by means of firmware).

[0133] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0134] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0135] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0136] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A motion adaptive method for robot vision language navigation, characterized in that: The method is used for a visual language navigation task with uncertain robot motion, and comprises the following steps: Step S101, determining the numerical range of the robot's movable action space and generating random actions; Step S102, based on the changes obtained by the interaction between the random action and the environment, the robot action is used to influence the self-encoding module to obtain an action embedding vector; Step S103, based on the action embedding vector, compare the specific actions in the data set and calculate the difference to determine the action currently executed by the robot and the semantic meaning corresponding to the action; Step S104, based on the interaction between the current execution action and the environment, a cross-modal attention feature extraction module is used to obtain multimodal features of the robot visual language; Step S105, using the robot's visual language multimodal features as the diffusion strategy decision module conditions, iteratively inferring the robot's next execution action, and updating the robot's state information; Step S106, repeating steps S102 to S105 until the robot reaches the vicinity of the target, thus ending the navigation task; The changes obtained based on the interaction between random actions and the environment are specifically: the robot interacts with the visual language navigation environment by performing random actions, and obtains first-person perspective observation pictures before and after the robot performs the current action; The method of using the robot action to influence the self-encoding module to obtain the action embedding vector includes: First, observe the first-person perspective images o of the robot before and after performing the current action. t-1 and t , perform pre-trained ResNet50 network visual encoding and linear transformation dimension processing to obtain the extracted visual feature r t ; Then, the robot currently performs action a t-1 =a i Index to obtain the memory vector matrix corresponding to a i The associated latest memory vector m i ; Finally, the memory vector m i and visual features r t Passed to the recurrent neural network, the update is done by action a i The action embedding vector e that causes the state change i ; Determining the current action performed by the robot and the semantic meaning of the action includes: First, the relevant actions in the specific discrete action space in the data set are indexed to obtain the corresponding action embedding vectors in the action embedding vector matrix; Then, the difference between the action embedding vector of the robot's current action and the action embedding vector of the corresponding action in the specific discrete action space in the data set is calculated by cosine similarity, and the semantic meaning of the current action is obtained by comparison, that is, the specific value of the forward step length or turning angle of the robot's current action; Finally, these values ​​are used as fixed measurement values ​​for the relevant actions in the discrete action space of the robot in this visual language navigation task.

2. The motion adaptation method for robot vision language navigation according to claim 1 is characterized in that: The numerical range of the robot's movable action space includes a numerical range of forward step length and a numerical range of turning angle.

3. The motion adaptation method for robot vision language navigation according to claim 1, characterized in that: The cross-modal attention feature extraction module is used to obtain the robot visual language multimodal features, including: First, the robot interacts with the environment based on deterministic actions to obtain the first-person perspective of the robot. ot and natural language navigation trajectory instructions; Then, for the RGB frame ov t Observation values, using pre-trained ResNet50 to collect semantic visual spatial features V t = {v i }, where i represents the spatial position index, and then average pooling is performed; for the depth frame od t observations, using a modified ResNet50 trained to perform point target navigation and represent these features D t =[d1, ..., d wh ], where w represents the image width and h represents the image height; the natural language navigation trajectory instructions are converted into corresponding GLoVE embeddings, and features are extracted using a bidirectional long short-term memory network; Finally, a Transformer-based cross-modal attention feature extraction model is used to extract visual-linguistic multimodal features between visual images and language instructions, where the visual-linguistic multimodal features include: natural language instruction deep features Visual RGB-Natural Language Command Features Visual Depth-Natural Language Command Features And the hidden layer information h of the GRU network t , specifically: Among them, Attn represents the scaled dot product attention mechanism, Represents the natural language instruction feature S and the hidden layer information h of the GRU network t The deep features of natural language instructions extracted after the cross-attention mechanism, Represents the visual RGB feature V t Deep features with natural language instructions The visual RGB-natural language command features extracted after the cross-attention mechanism, Represents the visual depth feature D t Deep features with natural language instructions Visual depth-natural language command features extracted after cross-attention mechanism.

4. The motion adaptation method for robot vision language navigation according to claim 1, characterized in that: The method of using the robot visual language multimodal features as the conditions of the diffusion strategy decision module and iteratively reasoning the next action of the robot is as follows: using the robot visual language multimodal features as the conditions of the diffusion strategy decision module and using Transformer to generate the conditional probability distribution p(a t |c t ), the diffusion strategy starts from the sampled noise action A t (k) Step by step iterative denoising until a noise-free action A is restored iteratively t (0) is the next execution action of the robot, where the iterative denoising process is: x k -1=α(x k -ge θ (x k ,k)+N(0,σ 2 I)) Among them, α, γ, σ represent the correlated noise learning rate, ε θ represents the noise prediction network, N(0,σ 2 I) represents the Gaussian noise added at each iteration, x k Represents the k-th iteration denoising action.

5. The motion adaptation method for robot vision language navigation according to claim 1, characterized in that: The robot state information includes a first-person-view observation picture of the robot and position information, wherein the first-person-view observation picture includes an RGB frame and a depth frame.

6. A system using the motion adaptation method of robot vision language navigation according to claim 1, characterized in that: The system comprises a feasible space numerical range determination module (10), an action impact self-encoding module (20), an action metric value solution module (30), a cross-modal attention feature extraction module (40) and a diffusion strategy decision module (50); The feasible space numerical range determination module (10) is used to determine the measurement value range of the robot's executable actions in the visual language navigation task and provide a basis for random value selection; The action impact self-encoding module (20) is used to obtain the impact result of the robot's current action based on the interaction between the random action and the environment, and obtain the action embedding vector of the robot's current action in the visual language navigation task; The action metric value solving module (30) is used to compare the action embedding vector of the currently executed action with the corresponding action in the specific discrete action space in the data set, calculate the difference between the two vectors by cosine similarity, obtain the semantic meaning corresponding to the currently executed action, and thus determine the metric value of the action in the new discrete action space of the robot in this navigation task; The cross-modal attention feature extraction module (40) is used to extract visual language action multimodal features based on the robot first-person perspective observation picture and natural language navigation trajectory instructions obtained by interacting with the environment through deterministic actions, so as to obtain robot visual language multimodal features; The diffusion strategy decision module (50) is used to perform conditional denoising diffusion based on the multimodal features of the robot visual language to iteratively restore the sampled noise action into a noise-free action as the next execution action of the robot.

7. The system according to claim 6, characterized in that The action impact self-encoding module (20) comprises a visual feature extraction unit (21) and an action embedding vector generation unit (22) connected in sequence; The visual feature extraction unit (21) is used to obtain the first-person perspective observation pictures of the robot before and after performing the current action based on the interaction between the random action and the environment. t-1 and t , perform pre-trained ResNet50 network visual encoding and linear transformation dimension processing to obtain the extracted visual feature r t ; The action embedding vector generating unit (22) is used to generate an action embedding vector based on the current action a performed by the robot. t-1 =a i Index to obtain the memory vector matrix corresponding to a i The latest memory associated with m i , the memory vector m i and visual features r t Passed to the recurrent neural network, the update is done by action a i The action embedding vector e that causes the state change i .

Images