A UAV visual language navigation method based on large model task parsing

Abstract

The present invention provides a visual language navigation method for unmanned aerial vehicles based on large model task analysis, comprising the following steps: S1, using a navigation task analysis module to decompose navigation instructions into multiple subtasks, and obtaining text features of the currently executed subtask through a subtask switching module after passing through a text encoder; S2, using a visual encoder to obtain visual features of a forward-looking image input for the current subtask; S3, using visual text features to obtain navigation action outputs corresponding to each moment of the current subtask through a multimodal encoder. The present invention can improve the interpretability and accuracy of visual language navigation action decisions.

Description

Technical Field

[0001] The present invention relates to the fields of embodied intelligence, multimodality and navigation, and in particular to a visual language navigation method for unmanned aerial vehicles based on large model task analysis. Background Art

[0002] Visual language navigation refers to humans issuing instructions to intelligent agents in the form of natural language. The intelligent agent will comprehensively understand the received instruction information and the visual information obtained from the environment, and perform corresponding actions in the environment to complete the tasks required by the instructions. As a cross-disciplinary direction, visual language navigation has both scientific research value and practical application value, making its exploration gradually become a hot issue.

[0003] In the current visual language navigation methods, in-depth analysis of navigation text to refine navigation instructions is the key direction to promote research. Research focuses include efficient parsing of complex semantic instructions, designing auxiliary reasoning tasks to optimize navigation performance, and using large language models and advanced natural language processing technologies to enhance system capabilities. Traditional grammar-based methods use formal grammatical rules and vocabularies to represent instructions, which is also known as the semantic parsing problem. For example, by using context-free grammars or compositional category grammars, researchers map natural language instructions to executable navigation plans. However, these methods often require expensive engineering design and predicate ontologies. With the rapid development of artificial intelligence, large language models such as BERT and GPT-3 have shown excellent performance, especially in the fields of text generation, machine translation, and dialogue systems. These models can capture the deep semantic features of language through pre-training on large-scale datasets, providing a strong language understanding foundation for visual language navigation tasks. Furthermore, hierarchical prompt systems such as "thinking chain prompts" and problem-solving "thinking trees" improve the quality of the model's response to specific problems by guiding the model to think more deeply and accurately. In addition, large language models such as GPT-4 can generate high-quality text based on given prompt words. Based on various navigation frameworks such as CorNav, ADAPT, and OVER-NAV, they can extract key information from natural language instructions and derive the target object corresponding to the landmark information through self-correction planning capabilities or modality-aligned action prompts to achieve precise navigation. These methods not only improve the accuracy of the intelligent agent's understanding of navigation instructions, but also enhance its autonomous navigation capabilities in complex environments.

[0004] In the analysis of navigation text for UAV visual language navigation, due to the large visual observation range of UAV, the description of the target in the navigation instructions will be more specific and complex. At the same time, due to the lack of intermediate supervision, it is impossible to evaluate the execution of each part of the instruction by the UAV during the navigation process. The existing keyword extraction based on large models is too rough to achieve accurate description of the target, which affects the subsequent target positioning in visual observation. Therefore, it is very important to adopt a unified task parsing paradigm to provide fine-grained annotations for UAVs during the training process. Summary of the invention

[0005] In order to solve the problem of difficulty in unifying task parsing due to a broader description of the airspace environment, the present invention provides a UAV visual language navigation method based on large-model task parsing with the goal of achieving detailed reconstruction of complex navigation instructions.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme:

[0007] A UAV visual language navigation method based on large model task parsing includes the following steps:

[0008] S1. Decompose the navigation instruction into multiple subtasks by using the navigation task parsing module, and obtain the text features of the currently executed subtask through the subtask switching module after passing through the text encoder;

[0009] S2, using the visual encoder to obtain the visual features of the front view image input for the current subtask;

[0010] S3. Use visual text features to obtain the navigation action output corresponding to each moment of the current subtask through a multimodal encoder.

[0011] Furthermore, the navigation task parsing module is specifically:

[0012] For the input original navigation instruction U, the system prompts are set to use the large language model to achieve fine-grained parsing of the entire navigation task. Among them, the design criteria of the system prompts mainly include the following points: 1) Each navigation subtask obtained by division corresponds to an independent and clear navigation behavior, which usually requires the drone to perform multiple operations to complete, such as "go straight", "turn left" or "find a specific object". 2) The target description related to the navigation behavior is mainly divided into phrases corresponding to the trigger target, tracking target and termination target. For navigation behaviors such as "take off" and "turn left", the subtasks only contain trigger targets and termination targets, while for long-term navigation behaviors such as "fly towards" and "fly over", additional tracking targets are used to guide the action output of the drone during the duration of the navigation behavior. 3) The key targets that appear in the entire navigation route will be transferred between different navigation subtasks, that is, the termination target of the current subtask is the trigger target of the next navigation subtask.

[0013] Finally, the text structure of each navigation subtask contains four phrases: navigation behavior, trigger target, tracking target, and termination target. The integrity and continuity of the entire navigation process can be guaranteed through a unified task parsing paradigm.

[0014] Furthermore, the subtask switching module is specifically:

[0015] Encoding the current state of the drone using a recurrent neural architecture , while considering the visual features of the current forward-looking image Text features of tokens corresponding to the terminal target Get the consistent positioning features of the visual language at the current moment , the formula is as follows:

[0016] ;

[0017] in, is the sigmoid activation function, and Represents the learnable weights, and then uses the unique hot encoding corresponding to the number of subtasks to be completed Calculate the switching probability of subtasks , the formula is as follows:

[0018] ;

[0019] in, and represents the learnable weights, and A learnable prior about when to switch is introduced. If the probability of switching exceeds a certain threshold, a shift signal is generated to read the next navigation subtask.

[0020] It should be noted that the subtask switching module enables the drone to follow complex navigation instructions more flexibly and accurately, and improves the drone's ability to understand instructions, making the entire navigation process more controllable and predictable.

[0021] Beneficial effects:

[0022] 1. From the perspective of model effect, the core of the present invention is to realize fine-grained navigation task analysis based on a large language model, and enhance the correlation between navigation behavior and target by dividing the target description. The visual language navigation technology of the present invention solves the problem of difficulty in unifying task analysis caused by more general text descriptions, and has greatly improved the generalization ability compared with other visual language navigation technologies.

[0023] 2. From a technical theory perspective, existing visual language navigation methods generally process the entire navigation text. The present invention innovatively introduces a sub-task switching module to guide the drone to intelligently switch between sub-instructions, effectively enhancing its traceability during the navigation process and reducing the risk of getting lost in complex instructions.

[0024] 3. From the application perspective, the two highly versatile modules used in the present invention, namely the navigation task parsing module and the subtask switching module, have a high degree of modularity. This design allows the present invention to be easily embedded in the existing drone navigation system, thereby reducing the complexity and cost of technical integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of a UAV visual language navigation method based on large model task parsing according to an embodiment of the present invention;

[0026] Figure 2 Schematic diagram of a visual language navigation network model in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0028] like Figure 1 As shown, a UAV visual language navigation method based on large model task parsing in an embodiment of the present invention includes the following steps:

[0029] S1. Decompose the navigation instruction into multiple subtasks by using the navigation task parsing module, and obtain the text features of the currently executed subtask through the subtask switching module after passing through the text encoder;

[0030] In this embodiment, the UAV visual language navigation task requires the UAV to navigate in an unknown environment according to the navigation instructions in natural language, which requires not only that the UAV can understand the navigation instructions, but also that it can combine the navigation behavior and corresponding targets mentioned in the instructions with the visual information in the actual environment. However, most existing methods are based on the original long and complex navigation instructions, and realize the tracking and operation of the UAV on the described navigation path while measuring the navigation progress at the word level, which greatly increases the difficulty of aligning the visual and language modalities.

[0031] In order to prevent the drone from being disturbed by other parts of the navigation instructions that have been completed or are to be completed, the navigation task parsing module proposed in this embodiment improves the accuracy and efficiency of navigation by forcing the drone to focus on the most relevant part of the navigation instructions, namely the current subtask. At the same time, this embodiment uses the designed subtask switching module to infer whether the current navigation subtask is completed and needs to switch to the next subtask to achieve a smooth transition of the entire navigation process.

[0032] When dividing long and complex navigation instructions into multiple short and easy-to-understand navigation subtasks, this embodiment considers that the relationship between navigation behavior and corresponding targets also plays a vital role in the action decision of the UAV, so these relationships are classified according to the role of the targets in the navigation behavior. For example, the target that instructs the UAV to start executing the navigation behavior is defined as the trigger target; the target that the UAV needs to continuously track during the execution of the navigation behavior is defined as the tracking target; the target that marks the completion of the navigation behavior is defined as the termination target. This clear definition helps the UAV better understand and execute complex navigation instructions.

[0033] In this embodiment, S1 specifically includes:

[0034] S1-1: The implementation process of the navigation task parsing module includes: for the input original navigation instruction U, a large language model is used to implement fine-grained parsing of the entire navigation task through the set system prompts.

[0035] Among them, the design criteria prompted by the system mainly include the following points:

[0036] 1) Each navigation subtask obtained by division corresponds to an independent and clear navigation behavior, which usually requires the drone to perform multiple operations to complete, such as "go straight", "turn left" or "find a specific object".

[0037] 2) The target descriptions related to navigation behaviors are mainly divided into phrases corresponding to trigger targets, tracking targets, and termination targets. For navigation behaviors such as "take off" and "turn left", their subtasks only include trigger targets and termination targets, while for long-term navigation behaviors such as "fly towards" and "fly over", additional tracking targets are used to guide the action output of the drone during the duration of the navigation behavior.

[0038] 3) Due to the complexity of the aerial navigation environment, the descriptions of key targets, i.e., those specific targets that have a decisive influence on the execution of UAV missions during navigation, are often relatively broad. In order to clearly define these key targets, it is usually necessary to use other auxiliary targets to highlight their uniqueness and characteristics. For example, "the blue building that on front of the green building" uses the presence of the green building in the scene to more accurately locate the key target of the blue building, thereby effectively guiding the aerial navigation behavior of the UAV. In addition, the key targets that appear in the entire navigation route will be transferred between different navigation subtasks, that is, the termination target of the current subtask is the trigger target of the next navigation subtask. Finally, the text structure of each navigation subtask contains four phrases: navigation behavior, trigger target, tracking target, and termination target. The integrity and continuity of the entire navigation process can be guaranteed through a unified task parsing paradigm.

[0039] S1-2: Use WordPieces to obtain the tokens sequence corresponding to the current navigation subtask and record it as Then the sentence start identifier [CLS], navigation instruction W, and segmentation identifier [SEP] are input into the pre-trained BERT network, and finally the text features of all L+2 tokens are obtained. . Where L represents the length of the navigation instruction. Represents the i-th word in the navigation instruction.

[0040] S1-3: After completing the current navigation subtask, the next navigation subtask is loaded through the subtask switching module. The specific implementation process includes: first, the current state of the drone is encoded through the recurrent neural structure , while considering the visual features of the current forward-looking image Text features of tokens corresponding to the terminal target , obtain the consistent positioning features of the visual language at the current moment , the formula is as follows:

[0041] ;

[0042] in, is the sigmoid activation function, and Represents the learnable weights, and then uses the unique hot encoding corresponding to the number of subtasks to be completed Calculate the switching probability of subtasks , the formula is as follows:

[0043] ;

[0044] in, and represents the learnable weights, and A learnable prior about when to switch is introduced. If the probability of switching exceeds a certain threshold, a shift signal is generated to read the next navigation subtask.

[0045] It should be noted that the subtask switching module enables the drone to follow complex navigation instructions more flexibly and accurately, and improves the drone's ability to understand instructions, making the entire navigation process more controllable and predictable.

[0046] S2. Use the visual encoder to obtain the visual features of the current subtask input front view image, including:

[0047] The front view image received by the drone is represented as ,in Represents the forward-looking image corresponding to the navigation time t of the current subtask. Since the visual encoder ResNet-152 has better performance on the ImageNet 2012 classification dataset than other shallow ResNet models, the visual features extracted by it have good generalization ability and are easy to migrate to other fields. Therefore, the present invention uses the ResNet-152 pre-trained on Place365 to extract the corresponding visual features from the forward-looking image received by the current subtask .

[0048] S3. Using visual text features, obtain the navigation action output corresponding to each moment of the current subtask through a multimodal encoder, including:

[0049] like Figure 2 As shown, text features are obtained through text and visual encoders respectively. and visual features Finally, the navigation decision process of the drone needs to comprehensively analyze the two modal information input. In this embodiment, considering that the lengths of different modalities may be highly unbalanced, a two-stream transformer network structure is used. This structure can balance the importance of intra-modal and inter-modal relationships through model design.

[0050] For each layer of the multimodal encoder, the relationship between modalities is first learned through modal interaction, that is, visual-text cross-attention is performed on the visual modality to focus on the relevant textual information. Similarly, visual-text cross-attention is performed on the textual modality to focus on the relevant visual information. Then the relationship within the text and visual modalities is learned through modal self-attention. Finally, the multimodal encoder outputs the updated text features and visual features , and use the fully connected layer to transform the visual features Mapped to the action output at the corresponding moment.

[0051] The loss function used in model training As shown below:

[0052] ;

[0053] Among them, the subtask switching uses the binary cross entropy to minimize the estimated shift probability and the probability of true value shift The loss between them, and for action prediction, the cross entropy minimization is used to predict the navigation action and truth value navigation actions The loss between.

[0054] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A UAV visual language navigation method based on large model task parsing, characterized in that: The steps include: S1. Decompose the navigation instruction into multiple subtasks by using the navigation task parsing module, and obtain the text features of the currently executed subtask through the subtask switching module after passing through the text encoder; The navigation task parsing module implements: For the input original navigation instruction U, the system prompt is set to use the large language model to achieve fine-grained analysis of the entire navigation task; the design criteria of the system prompt include: 1) Each navigation subtask obtained by division corresponds to an independent and clear navigation behavior, which is completed by the UAV performing multiple operations; 2) Divide the goal descriptions related to navigation behavior into phrases corresponding to trigger goals, tracking goals, and termination goals; 3) The key targets appearing in the entire navigation route are transferred between different navigation subtasks, that is, the termination target of the current subtask is the trigger target of the next navigation subtask; Finally, the text structure of each navigation subtask includes four phrases: navigation behavior, trigger target, tracking target, and termination target. The integrity and continuity of the entire navigation process can be guaranteed through a unified task parsing paradigm. S2, using the visual encoder to obtain the visual features of the front view image input for the current subtask; S3. Use visual text features to obtain the navigation action output corresponding to each moment of the current subtask through a multimodal encoder.

2. According to the large model task parsing based unmanned aerial vehicle visual language navigation method of claim 1, it is characterized in that: In S2, the subtask switching module implements: Encoding the current state of the drone using a recurrent neural architecture , while considering the visual features of the current forward-looking image Text features of tokens corresponding to the terminal target , obtain the consistent positioning features of the visual language at the current moment , the formula is as follows: ； in, is the sigmoid activation function, and represents the learnable weight; Use the unique hot encoding corresponding to the number of subtasks to be completed Calculate the switching probability of subtasks , the formula is as follows: ； in, and Represents learnable weights, one-hot encoding A learnable prior about when to switch is introduced; if the probability of shifting exceeds a certain threshold, a shift signal is generated to read the next navigation subtask.

Images