Unmanned aerial vehicle visual language navigation method based on multi-modal perception Mama

By applying the multimodal perception Mamba model in drone navigation, combining language, vision and direction information, the problems of poor navigation effects and complex operations in the existing technology are solved, and efficient and autonomous drone navigation is achieved.

CN119063736AActive Publication Date: 2024-12-03INST OF AUTOMATION CHINESE ACAD OF SCI

Patent Information

Application Number
CN202411248358.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-12-03
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

Due to model limitations, the existing drone navigation algorithms have poor navigation effects, slow training speed, and complex operation, making it difficult to achieve autonomous drone flight.

Method used

The Mamba model based on multimodal perception is adopted, combining three modalities: language, vision and direction, and information fusion learning is carried out through the Mamba model, and a multimodal sharing optimization strategy is designed to enhance the discriminantity and navigation efficiency of the model.

Benefits of technology

It improves the efficiency and accuracy of drone navigation, realizes the integrity and efficiency of autonomous navigation tasks, and reduces the cost of manpower and material resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119063736A_ABST
    Figure CN119063736A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-modal sensing Mama unmanned aerial vehicle visual language navigation method, which comprises the following steps: modeling text instruction information, and extracting text semantic features to enable an agent to understand the context content of an instruction; modeling is carried out on a visual image captured by an unmanned aerial vehicle agent and a position direction, and visual semantic and direction semantic features are extracted, so that the unmanned aerial vehicle agent can perceive environment information; modeling a historical navigation track, and extracting historical track characteristics, so that an unmanned aerial vehicle agent can mine key knowledge from historical information; based on a Mama model, carrying out fusion learning on information of three different modes of text, vision and direction, and reasoning a navigation action; all the parts are integrated into a unified framework, and overall training of the model is carried out. According to the method, the model can capture key navigation clues of different modes, multi-mode fusion is promoted, and the navigation effect is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of visual language navigation, and in particular relates to a visual language navigation method for an unmanned aerial vehicle based on multimodal perception Mamba. Background Art

[0002] The rapid development of science and technology and the Internet has led to the widespread use of drones, whether in military, transportation, agriculture, photography, or performance. However, compared with ground robots, drones are more difficult to operate because of the impact of altitude. Therefore, someone needs to observe the status of the drone at all times and use a controller to control its movement. This not only requires a lot of manpower and material resources, but is also not friendly to some novices. Therefore, it has become an urgent need to develop an intelligent drone that can complete flight missions without control through human-machine dialogue communication.

[0003] Recently, good progress has been made in developing such drone navigation algorithms using traditional deep network models (RNN, Transformer). However, since the RNN model changes with the sequence length, excessive forgetting will occur, and it cannot be trained in parallel, resulting in low training speed; and the transformer has a computational complexity of the square of the sequence length and slow reasoning, so the navigation effect of current methods is unsatisfactory. The recent emergence of the Mamba model has broken these limitations. With its advantages of parallel training and linear computational complexity, it has become a powerful model to replace the transformer. However, there is no research on applying Mamba to drone visual language navigation. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a UAV visual language navigation method based on multimodal perception Mamba. It is proposed for the first time to apply the Mamba model to UAV visual language navigation, and enhance the text instructions of key semantics according to the characteristics of the navigation task. The language instructions contain complex semantic knowledge and contextual content, and landmark words and direction words are the semantic content that navigation focuses on and understands. Therefore, using landmark words and direction words to enhance the understanding of instructions plays an important role in promoting the fusion and alignment of subsequent modalities. The present invention understands the visual environment. On the one hand, in order to perceive historical instructions and promote the learning of global content and historical clues, visual tokens related to instructions are designed; on the other hand, in order to fully understand environmental information and promote the mining of spatial structure and semantic content, visual tokens unrelated to instructions are designed. The present invention sets up a multimodal perception Mamba. Simply splicing the three modes of language, vision, and direction into a sequence and inputting them into Mamba has a suboptimal effect. Taking into account the multimodal characteristics of the task, a multimodal sharing optimization strategy is proposed. In addition to ensuring the characteristics of each modality, the content of the other two modalities is shared to promote the fusion between the modalities, so that the model can pay attention to the current environment and position status based on discriminability, explore content clues related to it, and promote navigation efficiency.

[0005] To achieve the above object, the present invention adopts the following technical solution: The visual language navigation method of UAV based on multimodal perception Mamba includes the following steps: Step S1: Model the text instruction information and extract the key semantic features of the text so that the drone agent can understand the context of the instruction; Step S2: Modeling the visual images and position directions captured by the drone agent, extracting visual semantic and directional semantic features, so that it can perceive environmental information; Step S3: Model the historical trajectory of navigation and extract the historical trajectory features, so that the drone agent can mine key knowledge from the historical information; Step S4: Based on the Mamba model, information from three different modalities, namely text, vision and direction, is integrated and learned to infer navigation actions; Step S5: Integrate the above steps into a unified framework and perform overall training of the framework.

[0006] The beneficial effects of the present invention are: 1) This paper proposes a multi-modal shared Mamba model, which promotes the fusion learning between different modalities, enhances the discriminability of the model, and improves the efficiency of navigation; 2) The present invention designs additional enhancement strategies from the two modalities of language and vision, and integrates them with the Mamba model into an overall framework for optimization, which promotes the completeness of the navigation task and ensures navigation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a flow chart of the visual language navigation method of unmanned aerial vehicle based on multimodal perception Mamba of the present invention. DETAILED DESCRIPTION

[0008] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0009] The UAV visual language navigation method based on multimodal perception Mamba proposed in the present invention models language, vision and direction respectively, performs multimodal perception fusion through Mamba, and integrates them into a framework for learning and optimization.

[0010] like Figure 1 As shown, the visual language navigation method of the unmanned aerial vehicle based on multimodal perception Mamba of the present invention comprises the following steps: Step S1: Model the text instruction information and extract the key semantic features of the text so that the drone agent can understand the context of the instruction; Step S2: Modeling the visual images and position directions captured by the drone agent, extracting visual semantic and directional semantic features, so that it can perceive environmental information; Step S3: Model the historical trajectory of navigation and extract the historical trajectory features, so that the drone agent can mine key knowledge from the historical information; Step S4: Based on the Mamba model, information from three different modalities, namely text, vision and direction, is integrated and learned to infer navigation actions; Step S5: Integrate the above steps into a unified framework and perform overall training of the framework.

[0011] Specifically, the step S1 includes: Step S11: Considering that the navigation task is set to complete a sub-navigation track according to a dialogue instruction in the overall task. The specific overall task contains multiple dialogues. Given the current dialogue instruction = { , }, and previous historical dialogues = { , , ,......, , }, in, Indicates agent problem, Indicates a text instruction. n indicates the turn of the conversation.

[0012] First, based on the perceived historical conversation content, the global command semantic features are obtained. : ; Among them, Bert is a language feature extractor based on transform. Represents the output dimension.

[0013] In order to reduce calculation, the present invention only obtains the global instruction Token characteristics.

[0014] Step S12: current dialogue instruction = { , } are encoded separately, and the landmark words and direction words in the dialogue instructions play a key role in semantic understanding. Therefore, the language parser NLTK is used to extract landmark words , Direction words , using a multi-head crisscross attention network ( ) to enhance the semantics of the current instruction. Specifically: ; in, Represents the features extracted from the current round of dialogue instructions, L represents the length of the feature token sequence, and D represents the dimension of each token.

[0015] ; Among them, Embedding is the feature embedding layer, which extracts the corresponding word features.

[0016] ; here It represents the instruction features enhanced by the corresponding landmark words and direction words.

[0017] Specifically, step S2 includes the following steps: Step S21: Given the current RGB observation image , use the visual encoder DarkNet to extract visual features In order to learn semantic clues corresponding to the current environment from historical dialogues and promote multimodal perception, a global command query is used to obtain visual tokens related to the command. The detailed description is as follows: ; ; ; Among them, K is the number of tokens, which is 49, and D is the feature dimension, which is 769. FFN is a feedforward neural network that performs nonlinear transformation on the input. is the command feature of all current rounds of dialogue extracted in step S1, is a normalization operation, , is the feature transformation matrix. d is the feature dimension of each attention head.

[0018] On the other hand, in order to mine the detailed information of the environment, fine-grained modeling of visual features is performed, and maximum pooling is used to obtain visual tokens that are independent of instructions. , where C is the number of tokens, which is set to 36. In addition, in order to facilitate subsequent multimodal perception, the pooled visual features are also used and the current dialogue command features Query and obtain the command token related to the current vision .

[0019] Step S22: Given the current direction angle ,right Find sin, cos to get a two-dimensional direction code, and use a vector Map it to D dimension to obtain directional features : = [sin( ), cos( )] ; Get directional features After that, in order to perceive the token associated with it in the current instruction, it promotes the subsequent modal fusion. and Query and obtain the command token associated with the direction .

[0020] Specifically, step S3 includes: According to step S2, and in order to facilitate calculation and improve efficiency, the present invention only uses the visual features related to the instruction in each step And directional characteristics Reserve and build the historical memory pool H_v = { }、H_d ={ }.

[0021] Specifically, step S4 includes: After obtaining the features of each modality, the present invention uses the Mamba model to perform multimodal perception fusion and action prediction, and makes corresponding improvements for the navigation task. First, the theoretical basis of the state space model (SSM) and Mamba is accepted, and the present invention is explained on this basis. The specific steps are as follows: Step S41: Similar to RNN and CNN, SSM is a typical neural network model for processing time series. Inspired by continuous systems, SSM uses a hidden state Convert a one-dimensional function or sequence Mapping to output ( ), the transformation process is as follows: ; ; Among them, (A, B, C, D) are the parameters of the model. In order to apply the above process to discrete signals, the zero-order hold (ZOH) technique is usually used, using the step parameter Convert the parameters (A, B) of processing continuous signals into parameters ( , ), the specific process is as follows: ; ; ; Among them, exp represents the exponential function. C is the output parameter like A and B. I represents the identity matrix.

[0022] After discretization, the process of processing discrete signals becomes: ; ; Where k represents the time step, represents the hidden state at the current time step k, Represents the input of the current time, Indicates the current time output.

[0023] Step S42: Since the current SSM model is processing time series, the parameters (A, B, C, ) will not be updated over time, that is, the parameters (A, B, C, ) is fixed at all time steps, which makes the model unable to selectively distinguish important content. The importance of the entire sequence token is exactly the same. This makes the model perform poorly in some selective tasks. Mamba uses a selective scanning algorithm to set the parameters (B, C, ) is modeled as a function of the input x, dynamically dependent on the current input, so that the model can understand the more important content in the sequence and ignore irrelevant content. The specific operation is as follows: ; ; ; in, represents a linear layer with an output dimension of D. Similarly and It also represents the linear layer. R is a low dimension, and N represents the dimension of the hidden state. Then the corresponding ,in is the zero-order hold (ZOH) technique number. It played an important role in this process. Controls the current time step for the input The balance between how much to ignore and how much to pay attention to. Similar to the gating mechanism in RNN, the larger Will reset the status And focus on the current input , while the smaller The state is maintained and the current input is ignored. This enables the model to have a selection function, and through network parameter updates, it can select the key content in the sequence and ignore irrelevant content.

[0024] Step S43: Considering the multimodal characteristics of visual language navigation, the present invention designs a parameterized shared modality Strategy, mapping language, vision, and direction features to the same dimension and concatenating them into a sequence X = { ; ( ; }, on the one hand, using the original On the other hand, a sequential modal fusion mechanism is designed to construct a low-rank , according to the order of a single modality in the input sequence, the features of the other two modalities are integrated, so that the modalities can interactively share key navigation information, promote the model's selection mechanism, and mine key navigation clues. The specific operations are as follows: = ; = { ; ; }; ; ; ; in, It is modally shared , represents the parameter matrix constructed from visual and directional features, The parameter matrix constructed from instructions and directions, Parameter matrix constructed from commands and vision. Indicates the extension of the same dimension as the instruction feature. The representation is expanded to the same dimension as the visual features, It means expanding to the same dimension of features in the same direction, so as to keep the dimensions aligned when different modalities are fused. Through the above operations, different modalities of the model can share knowledge and promote the fusion between modalities. It is more discriminative, can fully consider information from different modalities, can pay more attention to navigation clues related to the current environment position status from history and text, and improve navigation efficiency.

[0025] Step S44: After optimizing the Mamba model for the visual language navigation task in step S43, the modal fusion output is obtained through the improved Mamba model, and then the corresponding action and attention area are predicted through the fully connected layer.

[0026] ; ( ; ; in, It is through the above Optimized multimodal encoder, is the predicted 3D position coordinate, It represents the predicted 1-dimensional process completion index, which determines the degree of process completion. When it is greater than a threshold, navigation stops. represents the attention mask, which is used to determine the area that the current drone is paying attention to. R is the size of the original image mapped to the predicted mask area. The loss function of the above dynamic process is as follows: ; ; , ; in, represents the action prediction loss, represents the prediction loss of the attention area, N represents the total number of attention areas, i represents the attention area of ​​each unit, represents the predicted value of the i-th area of ​​interest, , Represents the mean and variance of the view. represents the mean square error loss, Indicates the calculation of the rotation angle change, represents the normalized scan path score loss, , , , which means obtaining labels from real trajectories.

[0027] Specifically, step S5 includes: The integration of the unified framework is to optimize the following loss function L: + .

[0028] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A visual language navigation method for unmanned aerial vehicles based on multimodal perception Mamba, characterized in that: The following steps are involved: Step S1: Model the text instruction information and extract the key semantic features of the text so that the drone agent can understand the context of the instruction; Step S2: Modeling the visual images and position directions captured by the drone agent, extracting visual semantic and directional semantic features, so that it can perceive environmental information; Step S3: Model the historical trajectory of navigation and extract the historical trajectory features, so that the drone agent can mine key knowledge from the historical information; Step S4: Based on the Mamba model, information from three different modalities, namely text, vision and direction, is integrated and learned to infer navigation actions; Step S5: Integrate the above steps into a unified framework and perform overall training of the framework.

2. The method for unmanned aerial vehicle visual language navigation based on multimodal perception Mamba according to claim 1 is characterized in that: The step S1 comprises: Step S11: Given the current dialogue instruction = { , } and history dialogue commands = { , , ,......, , },in Indicates the proxy problem, Represents a text instruction, let W = { }, extract global text instruction features and the current instruction characteristics ; Step S12: In order to capture the key natural language navigation information in the instructions and promote the understanding of the agent text instructions and the learning of the context content, the landmark nouns in the current dialogue instructions are , directional prepositions Extract it and use the same encoder to obtain features , , using a multi-head cross attention network to obtain command features based on landmark words and direction words enhancement ; The natural language navigation information includes landmarks and direction words.

3. The method for unmanned aerial vehicle visual language navigation based on multimodal perception Mamba according to claim 1 is characterized in that: The step S2 comprises: Step S21: Given the visual image at the current time step t , the angle of direction , extract visual features and directional characteristics ; Step S22: For visual features , two strategies are adopted to extract key navigation clues; one uses global text instruction features Obtaining instruction-related visual features through multi-head cross attention and fusion , which is used to fully perceive the environmental information related to the instruction and promote multimodal fusion; the other is to perform maximum pooling operation on the visual features to obtain visual features that are not related to the instruction. ,and They complement each other and are used to fully perceive important environmental clues, maintain the integrity of the environment, and promote the understanding and learning of spatial structure information by UAV agents; Step S23: At the current time step t, the drone relies more on the current vision and direction to perceive relevant semantic clues from the command, promoting multi-modal fusion; the current visual features and directional characteristics Through two multi-head cross attention layers and the current dialogue command features Fusion and compression into a one-dimensional vector, to obtain and .

4. The method for visual language navigation of unmanned aerial vehicles based on multimodal perception Mamba according to claim 3 is characterized in that: The step S3 comprises: According to step S21 and step S22, only the visual features related to the instruction in each step are And directional characteristics Reserve and build the historical memory pool H_v = { }、H_d ={ }.

5. The method for unmanned aerial vehicle visual language navigation based on multimodal perception Mamba according to claim 1 is characterized in that: After the features of each modality are obtained in step S4, the Mamba model is used to perform multimodal perception fusion and action prediction, and corresponding improvements are made for the navigation task, including: Step S41: The state space model SSM is constructed by a hidden state Convert a one-dimensional function or sequence Mapping to output ( ), the transformation process is as follows: ; ; Among them, (A, B, C, D) are the parameters of the model. In order to apply the above process to discrete signals, the zero-order hold technique is adopted and the step size parameter is used. Convert the parameters (A, B) of processing continuous signals into parameters ( , ), the specific process is as follows: ; ; ; After discretization, the process of processing discrete signals becomes: ; ; Step S42: The Mamba model uses the selection scanning algorithm to convert the parameters (B, C, ) is modeled as a function of the input x, dynamically dependent on the current input, allowing the model to understand the more important content in the sequence while ignoring irrelevant content, including: ; ; ; Among them, D is the dimension of input x, R is a low dimension, and N represents the dimension of hidden state; then the corresponding ; Step S43: Considering the multimodal characteristics of visual language navigation, design parameterization of modality sharing Strategy, mapping language, vision, and direction features to the same dimension and concatenating them into a sequence X = { ;( ; }, on the one hand, using the original Construct a strategy to maintain the characteristics and stability of a single modality; on the other hand, design a sequential modal fusion mechanism to construct a low-rank , according to the order of a single modality in the input sequence, the features of the other two modalities are integrated, so that the modalities can interactively share key navigation information, promote the model selection mechanism, and mine key navigation clues, including: = ; = { ; ; }; ; ; ; in, Sequence length extension operation to keep the dimensions aligned when different modalities are fused; Step S44: After optimizing the Mamba model for the visual language navigation task in step S43, the modal fusion output is obtained through the improved Mamba model, and then the corresponding action and focus area are predicted through the fully connected layer, including: ; ( ; ; in, is the predicted 3D position coordinate, It represents the predicted 1-dimensional process completion index, which determines the degree of process completion. When it is greater than a threshold, navigation stops. represents the attention mask, which is used to determine the area that the current drone is paying attention to; R is the mapping of the predicted mask area to the original image size. The loss function of the above dynamic process is as follows: ; ; , ; in, represents the mean square error loss, Indicates the calculation of the rotation angle change. represents the normalized scan path score loss, , , Indicates that the label is obtained from the real trajectory.

6. The method for visual language navigation of unmanned aerial vehicles based on multimodal perception Mamba according to claim 5 is characterized in that: In step S5, the integration of the unified framework is to optimize the following loss function L: + 。

Citation Information

Patent Citations

  • Visual language navigation method based on double semantic comprehension and fusion

    CN116429111A

  • Generating images from text based on hints

    CN117689745A

  • Visual language pair agent navigation system and method in continuous environment based on multiple visual features

    CN117935263A

  • Target detection method and device for multi-modal data

    CN118133191A

  • Target detection method and device, equipment and medium

    CN118196398A

Cited By

  • Method for automatically generating medical report by using multi-modal feature fusion Mamba

    CN119479969A

  • Visual language navigation method and system based on dynamic grid map

    CN120385352A

  • A visual language navigation method and system based on dynamic grid map

    CN120385352B

  • Robot operation track generation method based on structure perception and knowledge enhancement reasoning

    CN120765961A

  • Log anomaly detection method based on custom template and deep learning

    CN121051632A