A navigation waypoint prediction method for guiding continuous navigation behavior of robots

Abstract

This invention relates to a navigation waypoint prediction method for guiding continuous navigation behavior of a robot, comprising the following steps: extracting image feature vectors from panoramic RGBD images captured by the robot's camera; obtaining the robot's current offset angle and expanding the data dimension, then concatenating it with the image feature vectors to form corresponding action feature codes and panoramic image feature codes; aligning the robot's natural language commands with the panoramic image feature codes and orientation angle-related behavioral feature codes, respectively; performing cross-modal attention fusion on the encoded and aligned natural language commands at the decoder end with the command semantics after applying behavioral and visual attention, outputting a panoramic orientation selection prediction result; and predicting the navigation waypoint positions of sub-targets in the local navigation grid based on the panoramic orientation selection prediction result for the single-view RGBD image. Compared with existing technologies, this invention has advantages such as a close correlation between waypoint prediction and the navigation process, and strong model generalization.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and natural language generation technology, and in particular to a navigation waypoint prediction method for guiding continuous navigation behavior of robots. Background Technology

[0002] With the popularization and development of deep learning, visual language navigation, as an advanced and complex artificial intelligence task combining natural language understanding and environmental visual understanding, has the conditions for application and a wide range of use cases. The goal of this task is to issue a series of path instructions to the robot using natural language. The robot analyzes and judges the action decision based on language understanding, understands its own environment based on visual images, and performs navigation actions in accordance with the instructions, ultimately reaching the target point.

[0003] However, existing technologies simplify the navigation environment to a topological navigation map, only considering the transmission of nodes on the map and ignoring the intermediate navigation process. But considering that robots face a completely continuous environment in real-world applications and use a low-level motion space, the visual changes in a single movement are small and the movement distance is long, making it difficult for robots to learn visual language navigation in continuous environments and to align instructions with environmental observation information. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a navigation waypoint prediction method for guiding the continuous navigation behavior of robots. This invention has the advantages of being closely related to the waypoint prediction and navigation process and having strong model generalization ability.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] This invention provides a navigation waypoint prediction method for guiding continuous navigation behavior of a robot, comprising the following steps:

[0007] S1: Extract the image feature vector of the panoramic RGBD image captured by the robot's camera;

[0008] S2: Obtain the robot's current offset angle and expand the data dimension through triangulation. Then, concatenate it with the image feature vector to form the corresponding motion feature code and panoramic image feature code.

[0009] S3: Align the robot's natural language commands with panoramic image feature encoding and orientation angle-related behavioral feature encoding respectively, so that the robot can focus on the more important visual information cues and more obvious directional guidance in the commands respectively.

[0010] S4: The encoded and aligned natural language instruction is fused with the instruction semantics after applying behavioral and visual attention at the decoder end through cross-modal attention fusion, and the panoramic direction selection prediction result is output. The panoramic direction selection prediction result includes the direction selection and whether to stop.

[0011] S5: Select the prediction result based on the panoramic direction, and predict the navigation waypoint position of the sub-target in the local navigation grid for the single-view RGBD image.

[0012] Preferably, in step S1, a deep convolutional neural network is used to extract image feature vectors from the robot camera. The panoramic RGBD image includes an RGB image and a depth image. The deep convolutional neural network model includes a ResNet-50 network pre-trained on ImageNet and a ResNet-50 network pre-trained on a target navigation dataset. The RGB image is used as the input to the ResNet-50 network pre-trained on ImageNet, and the depth image is used as the input to the ResNet-50 network pre-trained on the target navigation dataset. The convolutional pooling features obtained by forward inference after inputting the image into the network are used as feature vectors.

[0013] Preferably, the panoramic RGBD image comprises 12 RGBD sub-images with 30-degree differences in orientation angle at the same horizontal height, each image corresponding to an RGB feature vector and a depth feature vector.

[0014] Preferably, in step S2, the robot's offset angle is the offset angle of each sub-image of the panoramic image based on the current orientation towards the center. The offset angle is encoded using embedding, and the expression is:

[0015] r = [cosθ, sinθ] × 32

[0016] In the formula, The offset angle is encoded, where θ is the heading angle offset for each sub-image, and ×32 indicates that the trigonometric transformation is repeated 32 times to expand the data dimension.

[0017] Preferably, the panoramic image feature f is formed by stitching together the RGB image features, depth image features, and offset angle codes corresponding to all sub-images, specifically as follows:

[0018]

[0019] In the formula, For the RGB image features corresponding to all sub-images, The depth image features are the corresponding to all sub-images.

[0020] Preferably, a soft attention mechanism is used to align the robot's natural language commands with panoramic image feature encoding and orientation angle-related behavioral feature encoding, respectively;

[0021] The soft attention mechanism focuses on phrases in the instruction that are relevant to panoramic image features, and the resulting output semantic feature vector expression is as follows:

[0022]

[0023]

[0024] The expression for the output semantic feature vector of the soft attention mechanism, which focuses on phrases related to behavioral features in the instruction, is as follows:

[0025]

[0026]

[0027] In the formula, u j W is the embedding encoding vector of the j-th instruction word. V and W A It is a learnable weight of the attention mechanism, α t,j and δ t,j These are attention weights related to the current panoramic image features and those related to behavioral features, respectively. For semantic feature vectors related to panoramic image features, These are semantic feature vectors related to action features.

[0028] Preferably, in step S4, a Bi-LSTM recurrent neural network is used as the encoder to encode the aligned natural language instructions, a gated recurrent unit (GRU) is used as the decoder to decode the panoramic image features based on the previous state and action selection to obtain a global state vector, visually related hidden states are used to apply attention to the panoramic image features to obtain visually related action selection predictions, and behaviorally related hidden states are used to apply attention to behavioral features to obtain behaviorally related action selection predictions.

[0029] Preferably, in step S4, the formulas describing visually related action selection prediction and behavior-related action selection prediction are as follows:

[0030]

[0031]

[0032]

[0033]

[0034] In the formula, W p W s W q Wf These are all learnable matrices of the linear layers of the model. These represent the visually relevant latent state vector and the behaviorally relevant latent state vector, respectively. and These represent visual-related action selection prediction and behavior-related action selection prediction, respectively. and All are 13-dimensional prediction vectors, where the first to 12 vectors are probability scores for panoramic direction prediction, and the 13th vector is the probability score for stop signal prediction.

[0035] Preferably, in step S5, the prediction result is selected based on the panoramic direction to determine whether it is a stop signal. If it is, it indicates that navigation has ended; otherwise, the panoramic navigation direction for the next step is determined. The visual features in this direction are combined with the command features to perform classification prediction on the divided navigation grid to determine the location of waypoints.

[0036] Preferably, in step S5, the navigation grid divides the navigation area within a 30° field of view of a single view into 120 grids of 10×12, based on an observation angle of 3 degrees and a travel distance of 0.25m per unit.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) This invention decouples the continuous navigation process by introducing a waypoint prediction structure. Waypoint prediction provides an abstract prediction of navigation actions at a higher dimension, resulting in significant visual changes and high-efficiency navigation performance. It also reduces the global navigation state space, greatly improving the learning ability and navigation effect of the continuous visual-language navigation model. The waypoint prediction model uses the Focal Loss function to calculate the loss for direction selection prediction, allowing the model to focus more on samples that are difficult to learn. Waypoints generated step-by-step using this method have an intuitive and obvious guiding effect on the continuous navigation of robots in indoor environments, and the resulting trajectories have good command matching performance.

[0039] (2) This invention proposes a cross-modal fusion model that focuses on the description of vision and behavior in the instructions. By applying attention to behavioral features and panoramic image features respectively, it effectively integrates the robot's action behavior with environmental observation, improves the robot's understanding of instructions and environment, and enables the robot to achieve good language instruction and visual matching performance in unknown environments through existing knowledge, with strong generalization ability. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the method of the present invention.

[0041] Figure 2This is a detailed system flowchart of the method of the present invention.

[0042] Figure 3 This is a schematic diagram illustrating the different dimensions of information of the instructions that are considered during the navigation process in the method of the present invention. Detailed Implementation

[0043] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0044] refer to Figure 1 As shown, this embodiment provides a navigation waypoint prediction method for guiding continuous navigation behavior of a robot, including the following steps:

[0045] S1: Extract the image feature vector of the panoramic RGBD image captured by the robot's camera.

[0046] Specifically, a deep convolutional neural network (DNN) is used to extract feature vectors from images captured by the robot's camera. The RGBD image includes both RGB and depth images. The DNN model comprises a ResNet-50 network pre-trained on ImageNet and a ResNet-50 network pre-trained on a target navigation dataset. The RGB image serves as input to the ResNet-50 network pre-trained on ImageNet, and the depth image serves as input to the ResNet-50 network pre-trained on the target navigation dataset. The convolutional pooling features obtained from forward inference after inputting the image into the network are used as the feature vector. To represent RGB image features, using Represents depth image features.

[0047] As an optional implementation, the panoramic RGBD image captured by the robot camera includes 12 RGBD sub-images with 30-degree differences in orientation angle at the same horizontal height. Each image corresponds to an RGB feature vector and a depth feature vector.

[0048] S2: Obtain the robot's current offset angle, expand the data dimension through trigonometric transformation, and then concatenate it with the image feature vector to form the corresponding behavioral feature code and panoramic image feature code.

[0049] The robot's offset angle is defined for each sub-image of the panoramic image based on its current orientation towards the center. The offset angle is encoded using an embedding method, expressed as follows:

[0050] r = [cosθ, sinθ] × 32

[0051] In the formula, For offset angle encoding, θ represents the heading angle offset of each sub-image, and ×32 indicates that the triangulation is repeated 32 times to expand the data dimension. The offset angle reflects the behavioral characteristics of each candidate direction, and the panoramic image feature f is stitched together from the RGB image features, depth image features, and offset angle encoding of all sub-images.

[0052] The offset angle code is concatenated with the visual features to form a feature vector:

[0053] In the formula, The offset angle is encoded, where θ is the heading angle offset for each sub-image, and ×32 indicates that the trigonometric transformation is repeated 32 times to expand the data dimension.

[0054] S3: Soft attention is used to align the robot's natural language commands with panoramic image feature encoding and orientation angle-related behavioral feature encoding, so that the robot pays attention to the more important visual information cues and more obvious directional guidance in the commands.

[0055] The expression for the output semantic feature vector of the soft attention mechanism, which focuses on phrases in the instruction that are related to panoramic image features, is as follows:

[0056]

[0057]

[0058] The expression for the output semantic feature vector of the soft attention mechanism, which focuses on phrases related to behavioral features in the instruction, is as follows:

[0059]

[0060]

[0061] In the formula, u j W is the embedding encoding vector of the j-th instruction word. V and W A It is a learnable weight of the attention mechanism, α t,j and δ t,j These are attention weights related to the current panoramic image features and those related to behavioral features, respectively. For semantic feature vectors related to panoramic image features, This represents the semantic feature vector related to action features. Explicitly distinguishing attention allows the model to focus on the semantic meaning of instructions related to information in different dimensions.

[0062] S4: The natural language instruction is aligned using a sequence-to-sequence frame encoding. At the decoder, cross-modal attention fusion is performed with the instruction semantics after applying behavioral and visual attention. The output is a panoramic direction selection prediction result, which includes the direction selection and whether to stop.

[0063] A Bi-LSTM recurrent neural network is used as the encoder to encode natural language instructions, enabling bidirectional understanding of these instructions. A gated recurrent unit (GRU) is used as the decoder to decode panoramic image features based on the previous state and action selection, obtaining a global state vector. Visually relevant hidden states are applied to the panoramic image features to obtain visually relevant action selection predictions, and behaviorally relevant hidden states are applied to the behavioral features to obtain behaviorally relevant action selection predictions. The specific formulas are as follows:

[0064]

[0065]

[0066]

[0067]

[0068] In the formula, W p W s W q W f These are all learnable matrices of the linear layers of the model. These represent the visually relevant latent state vector and the behaviorally relevant latent state vector, respectively. and These represent visual-related action selection prediction and behavior-related action selection prediction, respectively. and All are 13-dimensional prediction vectors, where the first to 12 vectors are probability scores for panoramic direction prediction, and the 13th vector is the probability score for stop signal prediction.

[0069] An adaptive fusion model is used to fuse visually relevant action selection prediction and behavior-related action selection prediction. Taking the weight calculation of behavior-related prediction as an example, the output is adaptively weighted:

[0070]

[0071] In the formula, ω a W is the weight value for predicting behavior-related action choices. a and W v It is a learnable projection matrix.

[0072] The final probability distribution for the panoramic direction selection is, in other words, the probability distribution of the robot's predicted direction at the next moment:

[0073]

[0074] In the formula, This indicates the prediction result for the panoramic direction selection.

[0075] The Focal Loss function is used to calculate the difference between the predicted and true values:

[0076]

[0077] in, It is a one-hot encoding for selecting the truth direction.

[0078] S5: By using an attention mechanism and a classification network, the prediction results are selected based on the panoramic direction. The sub-target navigation waypoint positions are predicted in the local navigation grid for single-view RGBD images. The navigation process is decoupled into sub-target waypoint prediction and waypoint navigation, which improves the model's learning ability in continuous environments.

[0079] Specifically, based on the panoramic direction, the prediction result is selected to determine whether it is a stop signal. If so, it indicates that navigation has ended; otherwise, a navigation waypoint is determined and low-level navigation actions are performed to reach the navigation waypoint of the sub-target.

[0080] The navigation grid classification head combines visual features in the selected direction with command features and performs classification prediction on the divided navigation grid to determine the location of waypoints.

[0081] The navigation grid divides the navigation area within a 30° field of view for a selected direction into 120 grids of 10×12, based on an observation angle of 3 degrees and a travel distance of 0.25m. The classification and calculation process for the navigation grid is as follows:

[0082]

[0083]

[0084]

[0085] in This represents the instruction attention distribution, which applies attention to instruction features using the global navigation state as the query vector. This represents the single-view image features in the selected direction l. W represents the visual attention distribution that applies attention to single-view image features using the instruction attention distribution as the query vector. l The weight matrix is ​​a learnable fully connected component. For the output of the classification network, the polar coordinate representation (ρ,d) of the waypoints is obtained through inverse mesh operations:

[0086]

[0087]

[0088]

[0089] Where ρ represents the offset angle of the waypoint relative to the robot's current direction, ρ0 is the view offset angle value corresponding to the direction selection action, and d∈[0.25,3] represents the distance of the waypoint relative to the robot.

[0090] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A navigation waypoint prediction method for guiding continuous navigation behavior of a robot, characterized in that, Includes the following steps: S1: Extract the image feature vector of the panoramic RGBD image captured by the robot's camera; S2: After obtaining the robot's current offset angle and expanding the data dimension through triangulation, it is concatenated with the image feature vector to form the corresponding behavior feature code and panoramic image feature code. The robot's offset angle is the offset angle of each sub-image of the panoramic image based on the current orientation towards the center. S3: Align the robot's natural language commands with panoramic image feature encoding and orientation angle-related behavioral feature encoding respectively, so that the robot can focus on the more important visual information cues and more obvious directional guidance in the commands respectively. S4: The encoded and aligned natural language instruction is fused with the instruction semantics after applying behavioral and visual attention at the decoder end through cross-modal attention fusion, and the panoramic direction selection prediction result is output. The panoramic direction selection prediction result includes the direction selection and whether to stop. S5: Select the prediction result based on the panoramic direction, determine whether it is a stop signal, if yes, it indicates the end of navigation, otherwise determine the next panoramic navigation direction, use the visual features in this direction combined with the command features, perform classification prediction on the divided navigation grid to determine the location of waypoints. The panoramic RGBD image includes 12 RGBD sub-images with 30-degree differences in orientation angle at the same horizontal height. Each image corresponds to an RGB feature vector and a depth feature vector. A soft attention mechanism is used to align the robot's natural language commands with panoramic image feature encoding and orientation angle-related behavioral feature encoding, respectively. The soft attention mechanism focuses on phrases in the instruction that are relevant to panoramic image features, and the resulting output semantic feature vector expression is as follows: The expression for the output semantic feature vector of the soft attention mechanism, which focuses on phrases related to behavioral features in the instruction, is as follows: In the formula, It is the first j The embedding encoding vector of each instruction word, and These are the learnable weights of the attention mechanism. and These are attention weights related to the current panoramic image features and those related to behavioral features, respectively. For semantic feature vectors related to panoramic image features, These are semantic feature vectors related to action features; In step S4, a Bi-LSTM recurrent neural network is used as the encoder to encode the aligned natural language instructions. A gated recurrent unit (GRU) is used as the decoder to decode the panoramic image features based on the previous state and action selection to obtain a global state vector. Visually related hidden states are used to apply attention to the panoramic image features to obtain visually related action selection predictions. Behavior-related hidden states are used to apply attention to the behavior features to obtain behavior-related action selection predictions.

2. The navigation waypoint prediction method for guiding continuous navigation behavior of a robot according to claim 1, characterized in that, In step S1, a deep convolutional neural network is used to extract image feature vectors from the robot's camera. The deep convolutional neural network model includes a ResNet-50 network pre-trained on ImageNet and a ResNet-50 network pre-trained on the target navigation dataset. RGB images are used as inputs to the ResNet-50 network pre-trained on ImageNet, and depth images are used as inputs to the ResNet-50 network pre-trained on the target navigation dataset. The convolutional pooling features obtained by forward inference after inputting the images into the network are used as feature vectors.

3. The navigation waypoint prediction method for guiding continuous navigation behavior of a robot according to claim 1, characterized in that, In step S2, the offset angle is encoded using an embedding method, expressed as follows: In the formula, Encoding for offset angle, For the heading angle offset of each sub-map, This indicates that the trigonometric transformation is repeated 32 times to expand the data dimensions.

4. The navigation waypoint prediction method for guiding continuous navigation behavior of a robot according to claim 3, characterized in that, The panoramic image features It is composed of the RGB image features, depth image features, and offset angle codes corresponding to all sub-images, specifically... In the formula, For the RGB image features corresponding to all sub-images, The depth image features are the corresponding to all sub-images.

5. The navigation waypoint prediction method for guiding continuous navigation behavior of a robot according to claim 1, characterized in that, In S4, the formulas describing visually-related action selection prediction and behavior-related action selection prediction are as follows: In the formula, , , , These are all learnable matrices of the linear layers of the model. , These represent the visually relevant latent state vector and the behaviorally relevant latent state vector, respectively. and These represent visual-related action selection prediction and behavior-related action selection prediction, respectively. and All are 13-dimensional prediction vectors, where the first to 12 vectors are probability scores for panoramic direction prediction, and the 13th vector is the probability score for stop signal prediction.

6. The navigation waypoint prediction method for guiding continuous navigation behavior of a robot according to claim 1, characterized in that, In step S5, the navigation grid divides the navigation area within a 30° field of view of a single view, further dividing the navigation area of ​​the candidate direction into sections based on an observation angle of 3 degrees and a travel distance of 0.25m. There are 120 grids in total.

Images