A visual language navigation method based on dual semantic understanding and fusion

Abstract

The present invention relates to a visual language navigation method based on dual semantic understanding and fusion, comprising the following steps: obtaining guiding instructions and visual observation images; extracting image features of the visual observation images and item features in the images; extracting guiding instruction long text features based on a natural language encoding network, and extracting guiding keyword text features based on expert experience rules; obtaining item-level image enhancement features and guiding keyword-based text enhancement features to obtain dual semantically enhanced visual and language features, and inputting these features into a visual-language cross-fusion module to obtain cross-modal fusion features; based on the cross-modal fusion features, using an action prediction subnetwork module to output a navigation position prediction result, control the robot to move toward the predicted navigation position, and update the corresponding visual observation image. Compared with existing technologies, the present invention has the advantages of high action prediction accuracy and strong interpretability.

Description

Technical Field

[0001] The present invention relates to the field of visual language navigation, and in particular to a visual language navigation method based on dual semantic understanding and fusion. Background Art

[0002] The vision-language navigation task aims to enable a robot to autonomously reach its destination along a trajectory described by natural language instructions. It is characterized by the unknown and unstructured nature of the environment, and the machine's reliance on visual images and textual instructions for reasoning. This technology is one of the key means for achieving human-computer interaction in the field of artificial intelligence. However, previous methods have mostly ignored semantic-level features derived from vision and text, such as object features in images and directional and landmark phrases in text. Instead, they use only coarse, broad features as network input, which contain a large amount of interference and irrelevant variables, resulting in the predicted path often not matching the given described trajectory. Summary of the Invention

[0003] The purpose of this invention is to provide a visual language navigation method based on dual semantic understanding and fusion, while improving the guiding semantic level feature expression of visual and language dual modalities, improving the accuracy of action prediction, and enabling the robot to have more powerful multimodal high-dimensional semantic understanding capabilities and better complete visual language navigation tasks.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] A visual language navigation method based on dual semantic understanding and fusion includes the following steps:

[0006] Step 1: Obtain input natural language-based guiding instructions and visual observation images of the surrounding environment observed by the robot at its current position;

[0007] Step 2: Based on the image feature extraction network and the object target recognition network, the image features of the visual observation image and the object features in the image are extracted respectively;

[0008] Step 3: Extract the guiding instruction long text features based on the natural language encoding network, and extract the guiding keyword text features based on expert experience rules;

[0009] Step 4: Use the image-item fusion module and the text-keyword fusion module to obtain item-level image enhancement features and guiding keyword-based text enhancement features, respectively, to obtain dual semantically enhanced visual and language features;

[0010] Step 5: Input the dual semantically enhanced visual and language features into the visual-language cross-fusion module to obtain cross-modal fusion features and improve the alignment of visual observations and language features;

[0011] Step 6: Based on the cross-modal fusion features, the action prediction sub-network module is used to output the navigation position prediction result, control the robot to move towards the predicted navigation position, and update the corresponding visual observation image.

[0012] The guiding instruction is a continuous and complete sentence describing a navigable trajectory.

[0013] The visual observation image is an RGB image.

[0014] The implementation methods of the image feature extraction network include CLIP, ViT, ResNet, and MobileNet.

[0015] The object recognition network is implemented by Fast R-CNN and YOLO.

[0016] The step 2 is specifically as follows: based on the trained image feature extraction network F I For the visual observation image V I Encode based on the trained object target recognition network model F O The original features V of the object image in the visual observation image O Encode using one-hot encoding F E The coordinates V of the object in the image O，d 、The category of the item V O，c and item confidence V O，t Encode and comprehensively obtain the item characteristics. The calculation formula is as follows:

[0017]

[0018]

[0019]

[0020]

[0021] Where, represents the image features after feature extraction, Represents the item features after feature extraction.

[0022] The step 3 is specifically as follows: based on the natural language text encoder G I Extract the long text features T of guiding instructions I , and based on expert experience rule G P , extract the action class T contained in the long text A and road sign class T L Keyword information, using the GloVe method G SExtract the embedded code and the calculation formula is as follows:

[0023]

[0024] (T A , T L )=G P (T I )

[0025]

[0026] Where, and Represent long text features and keyword text features respectively.

[0027] The image-item fusion module uses a Transformer encoder to learn the correlation between multiple item features, aggregates multiple item features using average pooling or attention-weighted averaging, and then uses a gating structure to adaptively fuse image features and item features. The calculation formula is as follows:

[0028]

[0029]

[0030]

[0031]

[0032] In the formula, Trans represents the Transformer encoder, Aggr represents weighted aggregation, γ represents the Sigmoid function, Concat represents connecting two vectors in the feature dimension, and ⊙ represents the multiplication of matrix corresponding items. represents the image features after feature extraction, represents the item features after feature extraction, W V is a learnable parameter, Representing item-level image enhancement features;

[0033] The text-keyword fusion module is based on the gate structure to adapt to the long text features and keyword text features Fusion is performed to obtain text enhancement features based on guiding keywords The calculation formula is as follows:

[0034]

[0035]

[0036] Where W Tare learnable parameters.

[0037] The framework of the visual-language cross-fusion module is the LXMERT cross-modal fusion model, which is based on the superposition of multiple multi-head attention modules. During the navigation reasoning process, the visual input is always used as the query vector, and the text input is used as the key vector and value vector. The calculation formula is as follows:

[0038]

[0039] Where, is the cross-modal fusion feature, represents the image enhancement feature, Represents text enhancement features.

[0040] The action prediction sub-network module consists of a linear layer, an activation layer, a random dropout layer, a linear layer, and a LayerNorm normalization layer. The calculation formula is as follows:

[0041]

[0042]

[0043] Where δ represents the activation layer, W a1 and W a2 is a learnable parameter, To fuse features across modalities, Drop represents the random drop layer and LN represents the LayerNorm normalization layer. In the learning process of the action prediction sub-network, the cross entropy function is used as the loss function and optimized.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] (1) The present invention proposes a dual semantic enhancement concept, and designs an image-object fusion module and a text-keyword fusion module for images and language respectively. Through the steps of separate enhancement and joint fusion, the model's ability to extract, fuse and understand the key guiding semantic features of vision and language multimodality can be effectively improved, thereby further improving the accuracy of action prediction.

[0046] (2) The present invention takes the characteristics of humans in solving visual language navigation tasks as its starting point, selects objects in images as key semantic features, and direction and signpost phrases in text as key guiding phrases, making the reasoning process of visual language navigation more practical and more interpretable and generalizable. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of the method of the present invention;

[0048] Figure 2A schematic diagram of image feature and text feature extraction in one embodiment;

[0049] Figure 3 Schematic diagram of the fusion module structure based on the gate control structure of the present invention. DETAILED DESCRIPTION

[0050] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0051] This embodiment provides a visual language navigation method based on dual semantic understanding and fusion, such as Figure 1 As shown, the following steps are included:

[0052] Step 1: Obtain input natural language-based guiding instructions and visual observation images of the surrounding environment observed by the robot at its current position.

[0053] The guiding instruction is a continuous, complete sentence describing a navigable trajectory. The visual image captured by the robot is an RGB image. If the robot is capable of observing a 360-degree panorama, the stored 360-degree panorama can be divided into 12 equal segments horizontally and 3 equal segments vertically, for a total of 36 sub-images.

[0054] Step 2: Based on the image feature extraction network and the object target recognition network, the image features of the visually observed image and the object features in the image are extracted respectively.

[0055] Based on trained image feature extraction networks such as CLIP, ViT, ResNet or MobileNet I For the visual observation image V I Encoding, based on the trained object target recognition network model F such as Fast R-CNN and YOLO O The original features V of the object image in the visual observation image O Encode using one-hot encoding F E The coordinates V of the object in the image O，d 、The category of the item V O，c and item confidence V o，t Encode and comprehensively obtain the item characteristics. The calculation formula is as follows:

[0056]

[0057]

[0058]

[0059]

[0060] Where, represents the image features after feature extraction, Represents the item features after feature extraction.

[0061] Generally, the dimension of feature extraction based on CLIP is 512, the dimension of feature extraction based on ViT is 768, and the dimension of feature extraction based on ResNet or MobileNet is 2048. To ensure that the vector dimensions are the same when adding, a learnable linear layer can be used to perform dimensionality reduction and reduction on inputs of different dimensions. To ensure that the numerical distribution of feature scalars is uniform, a normalization operation can be added at the end. A schematic diagram of a feature extraction method is shown below. Figure 2 shown.

[0062] Step 3: Extract the guiding instruction long text features based on the natural language encoding network, and extract the guiding keyword text features based on expert experience rules.

[0063] For example, in the instruction "Go forward and walk through the living room. Turn right when you see the piano with the flower," the guiding directional phrases include "Go forward," "walk through," and "Turn right," while the landmark phrases include "living room," "piano," and "flower." The original sentence contains rich contextual information, and the extracted guiding phrases highlight key information.

[0064] This embodiment is based on BERT or CLIP text encoder G I Large natural language pre-training models such as T are used to extract long text features of guiding instructions. I , and based on expert experience rule G P , extract the action class T contained in the long text A and road sign class T L Keyword information, using the GloVe method G S Extract the embedded code and the calculation formula is as follows:

[0065]

[0066] (T A , T L )=G P (T I )

[0067]

[0068] Where, and Represent long text features and keyword text features respectively.

[0069] Step 4: Use the image-item fusion module and the text-keyword fusion module to obtain item-level image enhancement features and guiding keyword-based text enhancement features, respectively, to obtain dual semantically enhanced visual and language features.

[0070] The image-item fusion module uses a Transformer encoder to learn the correlation between multiple item features, aggregates multiple item features using average pooling or attention-weighted averaging, and then uses a gating structure to adaptively fuse image features and item features. The calculation formula is as follows:

[0071]

[0072]

[0073]

[0074]

[0075] In the formula, Trans represents the Transformer encoder, Aggr represents weighted aggregation, γ represents the Sigmoid function, Concat represents connecting two vectors in the feature dimension, and ⊙ represents the multiplication of matrix corresponding items. represents the image features after feature extraction, represents the item features after feature extraction, W V is a learnable parameter, Represents item-level image enhancement features.

[0076] The text-keyword fusion module is based on the adaptive gating structure of long text features. and keyword text features Fusion is performed to obtain text enhancement features based on guiding keywords The calculation formula is as follows:

[0077]

[0078]

[0079] Where W T are learnable parameters.

[0080] Fusion modules based on gating structures such as Figure 3 shown.

[0081] Step 5: The dual semantically enhanced visual and language features are input into the visual-language cross-fusion module to obtain cross-modal fusion features and improve the alignment of visual observations and language features.

[0082] The framework of the vision-language cross-fusion module is the LXMERT cross-modal fusion model, which is based on the superposition of multiple multi-head attention modules. The calculation formula of the multi-head attention module is as follows:

[0083]

[0084]

[0085]

[0086]

[0087] φ(Q,K,V)=Concat(head1,head2,...,head n )W o

[0088]

[0089] Where Q, K, and V represent the query vector, key vector, and value vector, respectively, and W Q 、W K 、W V 、W O are all learnable variables.

[0090] During navigation reasoning, visual input is always used as the query vector, and text input is used as the key vector and value vector. The calculation formula is as follows:

[0091]

[0092] Where, is the cross-modal fusion feature, represents the image enhancement feature, Represents text enhancement features.

[0093] Step 6: Based on the cross-modal fusion features, the action prediction sub-network module is used to output the navigation position prediction result, control the robot to move towards the predicted navigation position, and update the corresponding visual observation image.

[0094] The action prediction sub-network module consists of a linear layer, an activation layer, a random dropout layer (Dropout), a linear layer, and a LayerNorm normalization layer (LN). The calculation formula is as follows:

[0095]

[0096]

[0097] In the formula, δ represents the activation layer, generally using the ReLU function, W a1 and W a2 is a learnable parameter, To fuse features across modalities, Drop represents the random drop layer and LN represents the LayerNorm normalization layer. In the learning process of the action prediction sub-network, the cross entropy function is used as the loss function and optimized.

[0098] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A visual language navigation method based on dual semantic understanding and fusion, characterized by: The following steps are involved: Step 1: Obtain input natural language-based guiding instructions and visual observation images of the surrounding environment observed by the robot at its current position; Step 2: Based on the image feature extraction network and the object target recognition network, the image features of the visual observation image and the object features in the image are extracted respectively; Step 3: Extract the guiding instruction long text features based on the natural language encoding network, and extract the guiding keyword text features based on expert experience rules; Step 4: Use the image-item fusion module and the text-keyword fusion module to obtain item-level image enhancement features and guiding keyword-based text enhancement features, respectively, to obtain dual semantically enhanced visual and language features; Step 5: Input the dual semantically enhanced visual and language features into the visual-language cross-fusion module to obtain cross-modal fusion features and improve the alignment of visual observations and language features; Step 6: Based on the cross-modal fusion features, the action prediction sub-network module is used to output the navigation position prediction result, control the robot to move towards the predicted navigation position, and update the corresponding visual observation image.

2. A visual language navigation method based on dual semantic understanding and fusion according to claim 1, characterized in that: The guiding instruction is a continuous and complete sentence describing a navigable trajectory.

3. The visual language navigation method based on dual semantic understanding and fusion according to claim 1 is characterized in that: The visual observation image is an RGB image.

4. The visual language navigation method based on dual semantic understanding and fusion according to claim 1 is characterized in that: The implementation methods of the image feature extraction network include CLIP, ViT, ResNet, and MobileNet.

5. The visual language navigation method based on dual semantic understanding and fusion according to claim 1 is characterized in that: The object recognition network is implemented by Fast R-CNN and YOLO.

6. The visual language navigation method based on dual semantic understanding and fusion according to claim 1 is characterized in that: The step 2 is specifically as follows: based on the trained image feature extraction network F I For the visual observation image V I Encode based on the trained object target recognition network model F O The original features V of the object image in the visual observation image O Encode using one-hot encoding F E The coordinates V of the object in the image O，d 、The category of the item V O，c and item confidence V O，t Encode and comprehensively obtain the item characteristics. The calculation formula is as follows: Where, represents the image features after feature extraction, Represents the item features after feature extraction.

7. The visual language navigation method based on dual semantic understanding and fusion according to claim 1 is characterized in that: The step 3 is specifically as follows: based on the natural language text encoder G I Extract the long text features T of guiding instructions I , and based on expert experience rule G P , extract the action class T contained in the long text A and road sign class T L Keyword information, using the GloVe method G S Extract the embedded code and the calculation formula is as follows: (T A ,T L )=G P (T I ) Where, and Represent long text features and keyword text features respectively.

8. The visual language navigation method based on dual semantic understanding and fusion according to claim 1 is characterized in that: The image-item fusion module uses a Transformer encoder to learn the correlation between multiple item features, aggregates multiple item features using average pooling or attention-weighted averaging, and then uses a gating structure to adaptively fuse image features and item features. The calculation formula is as follows: In the formula, Trans represents the Transformer encoder, Aggr represents weighted aggregation, γ represents the Sigmoid function, Concat represents connecting two vectors in the feature dimension, and ⊙ represents the multiplication of matrix corresponding items. represents the image features after feature extraction, Represents the item features after feature extraction, W V is a learnable parameter, Representing item-level image enhancement features; The text-keyword fusion module is based on the gate structure to adapt to the long text features and keyword text features Fusion is performed to obtain text enhancement features based on guiding keywords The calculation formula is as follows: Where W T are learnable parameters.

9. The visual language navigation method based on dual semantic understanding and fusion according to claim 1 is characterized in that: The framework of the visual-language cross-fusion module is the LXMERT cross-modal fusion model, which is based on the superposition of multiple multi-head attention modules. During the navigation reasoning process, the visual input is always used as the query vector, and the text input is used as the key vector and value vector. The calculation formula is as follows: Where, is the cross-modal fusion feature, represents the image enhancement feature, Represents text enhancement features.

10. The visual language navigation method based on dual semantic understanding and fusion according to claim 1, characterized in that: The action prediction sub-network module consists of a linear layer, an activation layer, a random dropout layer, a linear layer, and a LayerNorm normalization layer. The calculation formula is as follows: Where δ represents the activation layer, W a1 and W a2 is a learnable parameter, To fuse features across modalities, Drop represents the random drop layer and LN represents the LayerNorm normalization layer. In the learning process of the action prediction sub-network, the cross entropy function is used as the loss function and optimized.

Images