A visual language navigation method for unmanned aerial vehicles based on visual target reference guidance

Abstract

The invention provides a visual language navigation method for an unmanned aerial vehicle based on visual target reference guidance, comprising: using a navigation task parsing module to divide a navigation instruction into a plurality of subtasks, establishing references of different stages for each subtask, obtaining text features of a currently executed subtask through a subtask switching module after passing through a text encoder; identifying various target objects in a navigation scene at the current moment through a target detection network in a target positioning module, and obtaining visual features of a forward-looking image through a visual encoder; converting the referenced navigation text into a first-order logic program through a target parsing module, and inputting the program into a target positioning module in combination with the forward-looking image at the current moment to realize the positioning of a key target; using a visual guidance module to learn the current navigation behavior, and inputting the updated visual text features into a multimodal encoder to guide the output of navigation actions of the current subtask at each moment, and introducing an adaptive weighting mechanism during training to minimize the loss.

Description

Technical Field

[0001] The present invention relates to the fields of embodied intelligence, multimodality and navigation, and in particular to a visual language navigation method for an unmanned aerial vehicle based on visual target reference guidance. Background Art

[0002] In the field of visual language navigation, the main task of the embodied agent is to achieve effective navigation from the starting point to the target location based on the natural language instructions received. With the rapid development of artificial intelligence technology, visual language navigation has quickly become a hot topic of research, thanks to its profound significance in scientific exploration and its application potential in multiple industries. The application prospects of visual language navigation technology are broad, especially in the field of drones, which can enable drones to perform complex tasks more autonomously, such as search and rescue, environmental monitoring or agricultural surveys. In the future, as the technology matures further, we can foresee that visual language navigation will play a key role in more fields and bring more convenience and innovative solutions to human society.

[0003] At the forefront of visual language navigation research, the navigation behavior decision link is crucial for the autonomous navigation of intelligent agents. It requires the intelligent agent to make decisions based on the environmental information revealed by the visual input and the natural language navigation instructions. The key to achieving efficient navigation is for the intelligent agent to generate a series of accurate navigation actions based on the received natural language instructions and visual observation information. For different visual language navigation tasks, researchers have proposed mapping methods from features to action spaces using network structures such as long short-term memory networks (LSTM), graph neural networks (GNN), and transformers. LSTM is widely used in visual language navigation tasks due to its advantages in processing sequence data. These methods support the decision-making of intelligent agents in discrete environments by effectively capturing the long-term dependencies between historical observations and actions. Graph and transformer-based visual language navigation methods effectively capture the complex relationships between entities in the scene by constructing a graph representation of the environment or using the self-attention mechanism. These methods have jointly promoted the development of visual language navigation tasks.

[0004] In terms of behavioral decision-making for UAV visual language navigation, since the range of UAV actions in the air is wider and the dimension of action decision is higher, the behavioral complexity in navigation instructions when performing visual language navigation tasks is also higher. At present, most methods mainly rely on the similarity between the visual observation features at the current moment and the text features of the global context when performing action decisions, ignoring the dynamic changes of the visual targets corresponding to the key reference objects in the text in each navigation behavior. This visual guidance information can enable the model to have the ability to logically reason about navigation behaviors. Summary of the invention

[0005] Aiming at the problem that navigation actions are difficult to be autonomous due to more dimensions of large-scale actions, the present invention provides a UAV visual language navigation method based on visual target reference guidance, with the goal of realizing logical reasoning of navigation behavior by using a visual information guidance model.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A visual language navigation method for unmanned aerial vehicle based on visual target reference guidance comprises the following steps:

[0008] S1. Use the navigation task parsing module to divide the navigation instruction into multiple subtasks, and establish references for different stages for each subtask. After passing through the text encoder, the subtask switching module obtains the text features of the currently executed subtask;

[0009] S2, identifying various target objects in the current navigation scene through the target detection network in the target positioning module, and using the visual encoder to obtain the visual features of the front view image;

[0010] S3, converting the reference navigation text into a first-order logic program through a target parsing module, and inputting it into a target positioning module in combination with the current forward-looking image to realize the positioning of key targets;

[0011] S4. Use the visual guidance module to learn the current navigation behavior, and input the updated visual text features into the multimodal encoder to guide the output of the navigation action of the current subtask at each moment. An adaptive weighting mechanism is introduced during training to minimize the loss.

[0012] Further, in S4, the visual guidance module implements:

[0013] The specific position information of each reference object in the scene captured by the previous visual positioning module is encoded into absolute position features, and then combined with the visual features of the target to form the visual target tokens of these reference objects. Then, in order to make full use of the visual information of the reference objects, the text tokens corresponding to the navigation behavior in the current navigation subtask can fully perceive the changes in the spatial position of the reference objects in the scene, and thus logically infer the corresponding navigation decisions. We use a two-stream Transformer structure to perform cross-modal interaction and alignment on the text tokens and visual target tokens of the navigation behavior, so that the reliability of the generated navigation behavior decisions is further improved.

[0014] Further, in S4, the adaptive weighting mechanism is specifically:

[0015] Based on navigation text analysis and scene parsing, we first assess the difficulty of each subtask’s navigation behavior, and then achieve a comprehensive understanding of the difficulty of the current navigation behavior through complexity estimation based on visual observation and complexity estimation based on navigation behavior. Then, we design the weight coefficient corresponding to the loss function, and finally combine the model’s simulation with reinforcement learning for training.

[0016] The technical solution provided by the present invention enhances the logic and interpretability of navigation decisions through a visual guidance module based on a dual-stream Transformer network structure and an adaptive weighting mechanism, significantly improving the autonomous navigation capability and task execution efficiency of the UAV. The following are the beneficial effects described in detail:

[0017] 1. Enhanced spatial perception: The visual guidance module encodes the absolute position characteristics of reference objects, allowing the drone to accurately perceive the dynamic changes in the position of key targets in the environment, thereby improving the reliability of the drone's decision-making in complex environments.

[0018] 2. Personalized navigation strategy: The adaptive weighting mechanism dynamically adjusts the training weights according to the difficulty of each subtask and the complexity of visual observation, allowing the UAV to develop personalized navigation strategies for different tasks and improve the flexibility and adaptability of navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of a method for visual language navigation of an unmanned aerial vehicle based on visual target reference guidance according to an embodiment of the present invention;

[0020] Figure 2 Schematic diagram of a visual language navigation network model in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0022] like Figure 1 As shown, a visual language navigation method for an unmanned aerial vehicle based on visual target reference guidance according to an embodiment of the present invention comprises the following steps:

[0023] S1. Use the navigation task parsing module to divide the navigation instruction into multiple subtasks, and establish references for different stages for each subtask. After passing through the text encoder, the subtask switching module obtains the text features of the currently executed subtask, including:

[0024] For the original navigation instruction U provided, a detailed analysis of the entire navigation task is performed with the help of system prompts and a large language model with the goal of unifying the task description.

[0025] The construction of the system prompt follows the following principles: First, each derived navigation subtask should clearly correspond to an independent navigation behavior, which may involve the drone performing a series of operations, such as "flying in a straight line", "performing a left turn" or "locating a specific target". Secondly, the target text description related to the navigation behavior is subdivided into references for the three stages of triggering target, tracking target and termination target. For short actions such as "take off" and "turn left", the subtask will only include triggering and termination targets; for behaviors with longer duration such as "flying to" and "flying over", tracking targets are introduced to continuously guide the drone's actions during navigation. Finally, after establishing references for different stages of each navigation subtask, ensure that the termination target of the current subtask becomes the trigger point for starting the next subtask.

[0026] Then use WordPieces to get the tokens sequence corresponding to the current navigation subtask and record it as Then the sentence start identifier [CLS], navigation instruction W, and segmentation identifier [SEP] are input into the pre-trained BERT network, and finally the text features of all L+2 tokens are obtained. .

[0027] After completing the current navigation subtask, the subtask switching module will load the next navigation subtask. The specific implementation process includes: first, using the recurrent neural network structure to analyze the real-time state of the drone. At the same time, the visual features of the forward-looking image received by the drone at the current moment are comprehensively considered. Text features of tokens corresponding to the terminal target , thereby obtaining the consistent positioning features of the visual language at the current moment , the formula is as follows:

[0028] ;

[0029] in, is the sigmoid activation function, and Represents the learnable weights, and then uses the unique hot encoding corresponding to the number of subtasks to be completed Calculate the switching probability of subtasks , the formula is as follows:

[0030] ;

[0031] in, and represents the learnable weights, and A learnable prior about when to switch is introduced. If the probability of switching exceeds a certain threshold, a shift signal is generated to read the next navigation subtask.

[0032] S2, identifying various target objects in the current navigation scene through the target detection network in the target positioning module, and using the visual encoder to obtain the visual features of the front view image;

[0033] The forward-looking image that the UAV can receive at the navigation time t of the current subtask is expressed as , use Faster R-CNN to obtain all the target objects appearing in the current navigation scene for the forward-looking image at each moment. At the same time, because the visual encoder ResNet-152 has better results on the ImageNet 2012 classification dataset than other shallow ResNet models, the visual features extracted by it have good generalization ability and are easy to migrate to other fields. Therefore, the present invention uses the ResNet-152 pre-trained on Place365 to extract the corresponding visual features from the forward-looking image received by the current subtask .

[0034] S3. Convert the reference navigation text into a first-order logic program through the target parsing module, and input it into the target positioning module in combination with the current forward-looking image to realize the positioning of the key target, including:

[0035] First, in the target parsing module, the references corresponding to different stages of the current navigation subtask are used as input, and a pre-trained large language model is used to generate clear and unambiguous first-order logic programs. The generation of these programs relies on a system-level description of a set of common examples and first-order logic grammar rules, and the programs generated based on the references consist of a function name and corresponding parameters, and can be nested according to the logical hierarchy. The functions that constitute the first-order logic program are predefined and have built-in implementations, such as basic logical functions such as "and", "or" and "not", which are used to connect different propositions to form more complex logical structures. At the same time, there are functions such as "left" and "behind" to judge the relative positions between targets. More advanced functions involve various concepts about target descriptions in navigation scenarios.

[0036] Then, the first-order logic executor in the target positioning module is combined with the function in the logic program to distinguish the various visual targets detected in step S2 and construct the corresponding target hierarchy tree, which includes:

[0037] 1) Key targets: reference objects that appear in navigation instructions and match navigation behaviors, which can guide the action output of the drone;

[0038] 2) Target-based, reference objects that appear in navigation instructions to assist in determining the uniqueness of key targets;

[0039] 3) Other targets: objects that are not included in the navigation instructions but are detected in the forward-looking image.

[0040] For the navigation-related concept functions in the reference, the corresponding multi-layer perceptron MLP is initialized, that is, the relevant concepts in the specific field of navigation are implemented through modular neural networks to determine whether the target meets the conditions of the corresponding level. For example, given a navigation scene in which N targets are detected, the execution result of the concept function building(x) in the corresponding program can be represented as a vector of length N, and each element k in the vector corresponds to a Boolean value to indicate whether the target object k is defined as a building. Finally, the key targets in the text modality are connected with the corresponding target representations in the visual modality through the constructed hierarchical task tree, and the key target features that the drone can obtain at the navigation time t are represented as .

[0041] S4. Use the visual guidance module to learn the current navigation behavior, and input the updated visual text features into the multimodal encoder to guide the output of the navigation action of the current subtask at each moment. An adaptive weighting mechanism is introduced during training to minimize the loss.

[0042] In this embodiment, the visual language navigation of drones is essentially a very challenging and complex problem. Each navigation mode is actually achieved through a continuous and complete process, and most methods statically match the current visual target and the reference in the navigation text for each navigation moment. Therefore, it is particularly important to make the model perceive the changes of reference in visual observation to achieve logical reasoning of navigation behavior. In order to meet this challenge, the visual guidance module reveals the before-and-after reasoning relationship of the action sequence contained in each navigation behavior by utilizing the spatial position changes of key reference objects.

[0043] In this embodiment, the navigation behavior corresponding to each subtask has different levels of complexity. For complex subtasks such as flying over a target, the drone performs various navigation actions in three-dimensional space, which makes model learning more difficult. From the perspective of visual observation, if the reference object does not appear continuously in the field of vision during the execution of the drone's navigation behavior, it will also increase the difficulty of model learning. Therefore, it is more in line with the actual situation to use different loss weights for different navigation behaviors and visual observation results during training.

[0044] In this embodiment, S4 specifically includes:

[0045] S4-1: The implementation process of the visual guidance module includes: encoding the specific position information of the i-th reference object in the scene captured by the previous visual positioning module into an absolute position feature , combined with the visual features corresponding to the target Together they constitute the visual target token of the reference object and are recorded as Finally, the visual target tokens that the drone can obtain at navigation time t are expressed as The corresponding characteristic expression can be expressed as follows:

[0046] ;

[0047] ;

[0048] in, and Represents the coordinates of the upper left corner of the target detection box, and Represents the coordinates of the lower right corner of the target detection box, w and h represent the width and height of the target detection box, and W and H represent the width and height of the front view image. In order to make full use of the visual information of the reference object, so that the text tokens corresponding to the navigation behavior in the current navigation subtask can fully perceive the changes in the spatial position of the reference object in the scene, and thus logically infer the corresponding navigation decision, the present invention adopts a dual-stream Transformer structure to perform cross-modal interaction and alignment on the text tokens of the navigation behavior and the visual target tokens, and obtains a new text feature And the target characteristics This further improves the reliability of the generated navigation behavior decisions.

[0049] S4-2: Figure 2 As shown, for the text features obtained by the text encoder , visual features obtained by the visual encoder And the target features obtained by the target positioning module , the visual features and target features After stacking, it is regarded as a visual modality. The navigation decision process of the drone requires a comprehensive analysis of both text and visual modal information. In this embodiment, a two-stream transformer network structure is used as a multimodal encoder to output updated visual features. , and use the fully connected layer to update the visual features Mapped to the action output at the corresponding moment.

[0050] During model training, the action output of each navigation subtask uses cross entropy minimization to predict the navigation action. and truth value navigation actions The loss weights of different subtasks are evaluated from the perspectives of navigation behavior and visual complexity. The higher the complexity, the more difficult the model is to learn, and the higher the weight is given.

[0051] First, the complexity of navigation behavior is estimated. The true value of the drone's action output includes 8 dimensions, namely forward, left turn, right turn, left move, right move, rise, fall, and stop. The calculation of is as follows:

[0052] ;

[0053] in, represents one of the eight action truth values, Represents the probability of the true value of the action appearing in the current subtask.

[0054] Then, based on the complexity estimation of visual observation, the probability that the reference object appears in the forward-looking image is The higher the value, the lower the complexity. Direct use Combining the two complexity estimates and The weighting factor Get the loss weight corresponding to each subtask The calculation formula is as follows:

[0055] ;

[0056] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A visual language navigation method for unmanned aerial vehicles based on visual target reference guidance, characterized in that: The steps include: S1. Use the navigation task parsing module to divide the navigation instruction into multiple subtasks, and establish references for different stages for each subtask. After passing through the text encoder, the subtask switching module obtains the text features of the currently executed subtask; S2, identifying various target objects in the current navigation scene through the target detection network in the target positioning module, and using the visual encoder to obtain the visual features of the front view image; S3. Convert the reference navigation text into a first-order logic program through the target parsing module, and input it into the target positioning module in combination with the current forward-looking image to realize the positioning of the key target, including: First, in the target parsing module, the references of different stages corresponding to the current navigation subtask are used as input, and a clear and unambiguous first-order logic program is generated using the pre-trained large language model; then, the first-order logic executor in the target positioning module is combined with the functions in the logic program to distinguish the various visual targets detected in step S2 and construct the corresponding target hierarchy tree, which includes: 1) Key targets: reference objects that appear in navigation instructions and match navigation behaviors, which can guide the action output of the drone; 2) Target-based, reference objects that appear in navigation instructions to assist in determining the uniqueness of key targets; 3) Other targets: objects that did not appear in the navigation instructions but were detected in the forward-view image; For the navigation-related concept functions in the reference, the corresponding multi-layer perceptron MLP is initialized, that is, the relevant concepts in the specific field of navigation are implemented through modular neural networks to determine whether the target meets the conditions of the corresponding level; finally, the key targets in the text modality are connected with the corresponding target representations in the visual modality through the constructed hierarchical task tree, and the key target features that the drone can obtain at the navigation time t are expressed as ; S4. Use the visual guidance module to learn the current navigation behavior, and input the updated visual text features into the multimodal encoder to guide the output of the navigation action of the current subtask at each moment. An adaptive weighting mechanism is introduced during training to minimize the loss. The visual guidance module uses a dual-stream Transformer structure to perform cross-modal interaction and alignment on the text tokens and visual target tokens of the navigation behavior, so that the reliability of the generated navigation behavior decision is further improved.

2. The visual language navigation method for unmanned aerial vehicle based on visual target reference guidance according to claim 1 is characterized in that: In S4, the visual guidance module implements: By encoding the specific position information of the i-th reference object in the scene captured by the previous visual positioning module into an absolute position feature , combined with the visual features corresponding to the target Together they constitute the visual target token of the reference object and are recorded as , Then, in order to make full use of the visual information of the reference object, the text tokens corresponding to the navigation behavior in the current navigation subtask can fully perceive the changes in the spatial position of the reference object in the scene, and thus logically infer the corresponding navigation decision.

3. The method for unmanned aerial vehicle visual language navigation based on visual target reference guidance according to claim 1 is characterized in that: In S4, the adaptive weighting mechanism is specifically: Based on navigation text analysis and scene parsing, we first assess the difficulty of each subtask’s navigation behavior, and then achieve a comprehensive understanding of the difficulty of the current navigation behavior through complexity estimation based on visual observation and complexity estimation based on navigation behavior. Then, we design the weight coefficient corresponding to the loss function, and finally combine the model’s simulation with reinforcement learning for training.

Images