Unmanned aerial vehicle visual language navigation method based on multi-modal perception mamba

Abstract

The application discloses a kind of unmanned aerial vehicle visual language navigation methods of multimodal perception Mamba, comprising: text instruction information is modeled, text semantic features are extracted, so that agent can understand instruction context content;The visual image and position direction captured by unmanned aerial vehicle agent are modeled, and visual semantic and direction semantic features are extracted, so that it can perceive environmental information;The history trajectory of navigation is modeled, and history trajectory features are extracted, so that unmanned aerial vehicle agent can mine key knowledge from historical information;Text, vision and direction information of three different modalities are fused and learned based on Mamba model, and navigation action is reasoned;All the above parts are integrated into a unified framework, and the overall training of the model is carried out.The application enables the model to capture different modal key navigation clues, promotes multimodal fusion, and improves navigation effect.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of visual language navigation, and particularly relates to a UAV visual language navigation method based on a multi-modal perception Mamba. BACKGROUND

[0002] The rapid development of science and technology and the Internet has driven the wide application of unmanned aerial vehicles (UAVs). Whether it is military, transportation, agriculture, or photography and performance, the use of UAVs is involved. However, compared with ground robots, the operation of UAVs is more difficult because the influence of height needs to be considered. Therefore, a person needs to observe the state of the UAV at all times and use a controller to control the movement of the UAV. This not only requires a large amount of manpower and material resources, but is also not friendly to some beginners. Therefore, the development of an intelligent UAV that can complete a flight task through human-computer dialogue communication without control has become an urgent need.

[0003] Recently, good progress has been made in developing such a UAV navigation algorithm using traditional deep network models (RNN, Transformer). However, the RNN model changes with the sequence length, and over-fitting phenomenon occurs. In addition, the RNN model cannot be trained in parallel, resulting in low training speed. The Transformer is slow in inference because of the quadratic computational complexity of the sequence length. Therefore, the current method does not have satisfactory navigation effect. The emergence of the Mamba model breaks these limitations. With the advantages of parallel training and linear computational complexity, the Mamba model has become a powerful model to replace the Transformer. However, there is no research on applying the Mamba to UAV visual language navigation. SUMMARY

[0004] In order to solve the above technical problems, the present invention provides a UAV visual language navigation method based on multimodal perception Mamba. It is the first time to propose the application of the Mamba model to UAV visual language navigation, and enhance the key semantic text instructions based on the characteristics of the navigation task. The language instructions contain complex semantic knowledge and contextual content, and landmark words and direction words are the semantic content that navigation focuses on and understands. Therefore, using landmark words and direction words to enhance the understanding of instructions plays an important role in promoting the fusion and alignment of subsequent modalities. The present invention understands the visual environment. On the one hand, in order to perceive historical instructions and promote the learning of global content and historical clues, visual tokens related to instructions are designed; on the other hand, in order to fully understand environmental information and promote the mining of spatial structure and semantic content, visual tokens unrelated to instructions are designed. The present invention sets up a multimodal perception Mamba. Simply splicing the three modalities of language, vision, and direction into a sequence and inputting them into Mamba has suboptimal effects. Taking into account the multimodal characteristics of the task, a multimodal sharing optimization strategy is proposed. In addition to ensuring the characteristics of each modality, the content of the other two modalities is shared to promote the fusion of the modalities. This enables the model to focus on the current environment and location status based on its discriminability, explore related content clues, and improve navigation efficiency.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] The UAV visual language navigation method based on multimodal perception Mamba includes the following steps:

[0007] Step S1: Model the text instruction information and extract the key semantic features of the text so that the drone agent can understand the instruction context;

[0008] Step S2: Model the visual images and position directions captured by the drone agent, extract visual semantic and directional semantic features, and enable it to perceive environmental information;

[0009] Step S3: Model the historical trajectory of navigation and extract historical trajectory features, so that the drone agent can mine key knowledge from historical information;

[0010] Step S4: Based on the Mamba model, information from three different modalities, text, vision, and direction, is integrated and learned to infer navigation actions.

[0011] Step S5: Integrate the above steps into a unified framework and perform overall training of the framework.

[0012] The beneficial effects of the present invention are:

[0013] 1) The application proposes a multi-modal shared Mamba model, which promotes fusion learning between different modalities and enhances the discriminability of the model, improving the efficiency of navigation.

[0014] 2) The application designs additional enhancement strategies from language and visual modalities and integrates them with the Mamba model into an overall framework for optimization, promoting the completeness of the navigation task and ensuring navigation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a flowchart of the unmanned aerial vehicle visual language navigation method based on multi-modal perception Mamba of the application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical scheme and advantages of the application clearer, the application is further described in detail below in combination with specific embodiments and with reference to the drawings.

[0017] The unmanned aerial vehicle visual language navigation method based on multi-modal perception Mamba proposed by the application models language, vision and direction respectively and performs multi-modal perception fusion through Mamba, and learns and optimizes in an integrated framework.

[0018] As shown in Figure 1 The unmanned aerial vehicle visual language navigation method based on multi-modal perception Mamba of the application includes the following steps:

[0019] Step S1, model the text instruction information and extract the text key semantic features, so that the unmanned aerial vehicle agent can understand the instruction context content;

[0020] Step S2, model the visual image and position direction captured by the unmanned aerial vehicle agent and extract the visual semantic and direction semantic features, so that it can perceive environmental information;

[0021] Step S3, model the historical trajectory of navigation and extract the historical trajectory features, so that the unmanned aerial vehicle agent can mine key knowledge from historical information;

[0022] Step S4, fuse and learn the information of the three different modalities of text, vision and direction based on the Mamba model, and infer the navigation action;

[0023] Step S5, integrate the above steps into a unified framework and perform overall training of the framework.

[0024] Specifically, the step S1 includes:

[0025] Step S11, considering the setting of the navigation task is according to a certain dialogue instruction in the overall task, a segment of the sub-navigation trajectory is completed. The specific overall task contains multiple dialogues, and the current dialogue instruction is given = { , } and the previous historical dialogue = { , , ,......, , } wherein, represents the agent problem, represents the text instruction. n represents the round of the dialogue.

[0026] First, according to the perception of the historical dialogue content, the global instruction semantic feature :

[0027] ;

[0028] Wherein, Bert is a language feature extractor based on transform, represents the output dimension.

[0029] In order to reduce the calculation, the invention only obtains the token feature of the global instruction.

[0030] Step S12, the current dialogue instruction = { , } is encoded separately, and the landmark words and direction words in the dialogue instruction play a key role in semantic understanding. Therefore, the language parser NLTK is used to extract landmark words , direction words , and the current instruction is semantically enhanced using multi-head cross attention network ( ). Specifically:

[0031] ;

[0032] Wherein, represents the feature extracted by the current round dialogue instruction, L represents the length of the feature token sequence, and D represents the dimension of each token.

[0033] ;

[0034] Wherein, Embedding is a feature embedding layer, which extracts corresponding word features.

[0035] ;

[0036] Here represents the instruction feature enhanced by the corresponding landmark word and direction word.

[0037] Specifically, the step S2 comprises the following steps:

[0038] Step S21, given the current RGB observation image , extract visual features using the visual encoder DarkNet . In order to be able to learn semantic clues corresponding to the current environment from the historical dialogue, promote multi-modal perception, use global instruction query to obtain instruction-related visual token . The detailed description is as follows:

[0039] ;

[0040] ;

[0041] ;

[0042] wherein K is the number of tokens, 49, and D is the feature dimension, 769. FFN is a feedforward neural network that performs nonlinear transformation on the input. is the instruction feature of the current full round dialogue extracted in step S1, is a normalization operation, , is a feature transformation matrix. d is the feature dimension of each attention head.

[0043] On the other hand, in order to excavate the detailed information of the environment, the visual features are modeled in a fine-grained manner, and the instruction-independent visual token is obtained using maximum pooling, wherein C is the number of tokens, which is set to 36. In addition, in order to promote subsequent multi-modal perception, the visual features after pooling and the current dialogue instruction feature are also used to query, and the current visual-related instruction token is obtained.

[0044] Step S22, given the current direction angle , sin, cos to obtain a two-dimensional direction encoding, and use a vector to map it to D dimensions to obtain the direction feature :

[0045] = [sin( ), cos( ] ;

[0046] obtaining direction features After that, in order to perceive the token associated with it in the current instruction, promote subsequent modal fusion. Using and query, obtain the instruction token associated with the direction .

[0047] Specifically, the step S3 comprises:

[0048] According to step S2, and in order to facilitate calculation, improve efficiency, the present application only retains the visual features related to the instruction of each step and direction features , builds a history memory pool H_v ={ }, H_d ={ }.

[0049] Specifically, the step S4 comprises:

[0050] After obtaining the features of each modality, the present application uses the Mamba model for multi-modal perception fusion and action prediction, and makes corresponding improvements for navigation tasks. First, accept the theoretical basis of state space model (SSM) and Mamba, and on this basis, elaborate the present application, the specific steps are as follows:

[0051] Step S41: Similar to RNN, CNN, SSM is a typical neural network model for processing time series, inspired by continuous systems, SSM maps one-dimensional functions or sequences into output through a hidden state ( ), the transformation process is as follows:

[0052] ;

[0053] ;

[0054] Where (A, B, C, D) are the parameters of the model, in order to be able to apply the above process to discrete signals, usually use zero-order hold (ZOH) technology, using step parameter convert the parameters (A, B) for processing continuous signals into parameters (A, B) for processing discrete signals , , the specific process is as follows:

[0055] ;

[0056] ;

[0057] ;

[0058] where exp denotes the exponential function. C is the output parameter as A, B. I represents the identity matrix.

[0059] After the discretization process, the process of processing discrete signals becomes:

[0060] ;

[0061] ;

[0062] where k represents the time step, represents the hidden state at the current time step k, represents the input at the current time, represents the output at the current time.

[0063] Step S42: Since the current SSM model does not update the parameters (A, B, C, ) over time when processing time series, the parameters (A, B, C, ) are fixed at all time steps, which causes the model to be unable to selectively distinguish important content, and the importance of the entire sequence token is the same. This makes the model perform poorly in some selective tasks. Mamba uses a selective scanning algorithm to model the parameters (B, C, ) as a function of the input x, dynamically dependent on the current input, so that the model can understand more important content in the sequence and ignore irrelevant content. The specific operation is as follows:

[0064] ;

[0065] ;

[0066] ;

[0067] where, represents a linear layer with output dimension D, and and also represent linear layers. R is a low-dimensional, and N represents the dimension of the hidden state. Then, the corresponding is obtained through the discretization operation, where is the zero-order hold (ZOH) technique number. plays an important role in this process, controls the current time step to input Balance of how much to ignore or pay attention to. Similar to the gating mechanism in RNN, a larger will reset the state and pay attention to the current input , while a smaller will keep the state and ignore the current input. This allows the model to have a selection function, which can select the key content in the sequence and ignore irrelevant content through network parameter update.

[0068] Step S43: Considering the multi-modal characteristics of visual language navigation, the application designs a modal sharing parameterized strategy to map language, vision, and direction features to the same dimension and concatenate them into a sequence X = { ; ( ; }, On the one hand, the original construction strategy is used to maintain the features and stability of single modal; on the other hand, a sequential modal fusion mechanism is designed to construct a low-rank , which fuses the features of the other two modalities in the order of the input sequence single modal, so that the modalities can interact and share key navigation information, promote the selection mechanism of the model, and mine key navigation clues. The specific operation is as follows:

[0069] = ;

[0070] = { ; ; };

[0071] ;

[0072] ;

[0073] ;

[0074] Among them, is the modal sharing , represents the parameter matrix constructed by vision and direction features, the parameter matrix constructed by instructions and direction, the parameter matrix constructed by instructions and vision. represents the expansion to the same dimension as the instruction feature, represents the expansion to the same dimension as the vision feature, The dimension is extended to the same direction feature, so that the dimension alignment is maintained when different modalities are fused. Through the above operation, the model can share knowledge between different modalities, promote the fusion between modalities, and make the model more discriminative, can fully consider the information of different modalities, can pay more attention to the navigation clues related to the current environmental position state from history and text, and improve the efficiency of navigation.

[0075] Step S44: After the Mamba model is optimized for the visual language navigation task in step S43, the modal fusion output is obtained through the improved Mamba model, and then the corresponding action and attention area are predicted through the full connection layer.

[0076] ;

[0077] ;

[0078] ;

[0079] wherein, is the multi-modal encoder optimized above, is the predicted 3D position coordinates, represents the predicted 1D process completion index, which discriminates the degree of process completion, and the navigation stops when it is greater than a threshold. represents the attention mask, which is used to discriminate the area of the current unmanned vehicle attention. R is the mapping of the predicted mask area to the original image size. The loss function of the above dynamic process is as follows:

[0080] ;

[0081] ;

[0082] , ;

[0083] wherein, represents the action prediction loss, represents the attention area prediction loss, N represents the total number of attention areas, and i represents each unit attention area, represents the i-th attention area prediction value, , represents the mean and variance of the view. represents the mean square error loss, represents the calculation of the change of the rotation angle, represents the normalized scan path score loss, , , ​​, indicating that the label is obtained from the real trajectory.

[0084] Specifically, step S5 includes:

[0085] The integration of the unified framework is to optimize the following loss function L:

[0086] + .

[0087] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A UAV visual language navigation method based on multimodal perception Mamba, characterized by: The following steps are involved: Step S1: Model the text instruction information and extract the key semantic features of the text so that the drone agent can understand the instruction context; Step S2: Model the visual images and position directions captured by the drone agent, extract visual semantic and directional semantic features, and enable it to perceive environmental information; Step S3: Model the historical trajectory of navigation and extract historical trajectory features, so that the drone agent can mine key knowledge from historical information; Step S4: Based on the Mamba model, information from three different modalities, text, vision, and direction, is integrated and learned to infer navigation actions, including: Step S41: The state space model SSM is constructed by a hidden state Convert a one-dimensional function or sequence Mapping to output , , the transformation process is as follows: ； ； Among them, A, B, C, D are the parameters of the model. In order to apply the above process to discrete signals, the zero-order hold technique is adopted and the step size parameter is used. Convert the parameters A and B used to process continuous signals into parameters used to process discrete signals , The specific process is as follows: ； ； ； After discretization, the process of processing discrete signals becomes: ； ； Step S42: The Mamba model uses the selection scanning algorithm to set the parameters B, C, Modeling it as a function of the input x, dynamically dependent on the current input, enables the model to understand the more important content in the sequence while ignoring irrelevant content, including: ； ； ； Among them, D is the dimension of input x, R is a low dimension, and N represents the dimension of hidden state; then the corresponding ; Step S43: Considering the multimodal nature of visual language navigation, design parameterization of modality sharing Strategy, mapping language, vision, and direction features to the same dimension and concatenating them into a sequence X = { ;( ; }, on the one hand, using the original Construct a strategy to maintain the characteristics and stability of a single modality; on the other hand, design a sequential modal fusion mechanism to construct a low-rank The features of the other two modalities enable the interactive sharing of key navigation information between modalities, promote the model selection mechanism, and mine key navigation clues, including: = ； = { ; ; }； ； ； ； in, Sequence length extension operation to keep the dimensions aligned when fusing different modalities; Step S44: After optimizing the Mamba model for the visual language navigation task in step S43, the modal fusion output is obtained through the improved Mamba model, and then the corresponding action and attention area are predicted through the fully connected layer, including: ； ( ； ； in, is the predicted 3D position coordinate, Represents the predicted 1-dimensional process completion index, which determines the degree of process completion. When it is greater than a threshold, navigation stops. represents the attention mask, which is used to determine the area that the current drone is paying attention to; R is the mapping of the predicted mask area to the original image size. The loss function of the above dynamic process is as follows: ； ； , ； in, represents the mean square error loss, Indicates the calculation of the rotation angle change, represents the normalized scan path score loss, , , Indicates that the label is obtained from the real trajectory; Step S5: Integrate the above steps into a unified framework to perform overall training of the framework. The integration of the unified framework is to optimize the loss function L as follows: + 。 2. The UAV visual language navigation method based on multimodal perception Mamba according to claim 1 is characterized in that: The step S1 comprises: Step S11: Given the current dialogue instruction = { , } and history dialogue commands = { , , ,......, , },in Indicates the agency problem, Represents a text instruction, let W = { }, extract global text instruction features and current instruction characteristics ; Step S12: In order to capture the key natural language navigation information in the instructions and promote the understanding of the agent text instructions and the learning of the context content, the landmark nouns in the current dialogue instructions are converted into 、Directional prepositions Extract it and use the same encoder to obtain features 、 , using a multi-head cross attention network to obtain instruction features based on landmark words and direction words enhancement ; The natural language navigation information includes landmarks and direction words.

3. The UAV visual language navigation method based on multimodal perception Mamba according to claim 1 is characterized in that: The step S2 comprises: Step S21: Given the visual image at the current time step t , the angle of direction , extract visual features and directional characteristics ; Step S22: For visual features , two strategies are adopted to extract key navigation clues; one uses global text instruction features Obtain instruction-related visual features through multi-head cross attention and its fusion , which is used to fully perceive the environmental information related to the instruction and promote multimodal fusion; the other one performs the maximum pooling operation on the visual features to obtain visual features that are not related to the instruction. ,and They complement each other and are used to fully perceive important environmental clues, maintain the integrity of the environment, and promote the UAV agent's understanding and learning of spatial structure information; Step S23: At the current time step t, the UAV relies more on the current vision and direction to perceive the relevant semantic clues from the command, promoting the fusion of multimodal; the current visual features and directional characteristics Through two multi-head cross attention layers and the current dialogue instruction features Fusion and compression into a one-dimensional vector, to obtain and .

4. The UAV visual language navigation method based on multimodal perception Mamba according to claim 3 is characterized in that: The step S3 comprises: According to step S21 and step S22, only the visual features related to the instruction in each step are and directional characteristics Retain and build the history memory pool H_v ={ }、H_d ={ }.

Images