Unmanned aerial vehicle visual language navigation method based on target hierarchical tree

Abstract

The invention provides an unmanned aerial vehicle visual language navigation method based on a target hierarchical tree, which comprises the following steps: S1, acquiring a first-order logic program of a target in a navigation instruction through a target analysis module, and acquiring text features of the navigation instruction by using a text encoder; s2, constructing a hierarchical tree corresponding to various targets in the front-view image through a target positioning module, positioning a key target according to a first-order logic program of the target and obtaining a corresponding visual feature, and then extracting the visual feature of the front-view image by using a visual encoder; and S3, acquiring navigation action output corresponding to each navigation moment through a multi-mode encoder by utilizing the visual text features. According to the invention, the navigation decision quality is improved.

Description

Technical Field

[0001] The present invention relates to the fields of embodied intelligence, multimodality and navigation, and in particular to a visual language navigation method for unmanned aerial vehicles based on a target hierarchy tree. Background Art

[0002] Visual language navigation is an interdisciplinary research field that integrates cutting-edge technologies from multiple disciplines such as natural language processing, computer vision, and machine learning. The core challenge of visual language navigation is that the agent must be able to understand the complex instructions conveyed by humans through natural language and combine these instructions with visual information captured from the actual environment to make accurate navigation decisions. With the continuous advancement of artificial intelligence technology, visual language navigation has become a research hotspot. It not only has important scientific exploration significance, but also has broad application prospects.

[0003] In the current visual language navigation method, navigation scene perception is the key link to achieve autonomous navigation of intelligent agents. It requires the intelligent agent to understand the environment through visual input and make decisions in combination with natural language instructions. This not only involves the accurate interpretation of visual information by the intelligent agent, but also includes combining this information with navigation instructions to achieve a comprehensive understanding of the scene. For example, the intelligent agent identifies various types of targets in the current scene and matches them with the descriptions mentioned in the instructions. By adopting advanced visual language cross-modal alignment methods, various pre-trained agent tasks are used as a means to align the entities in the navigation instructions with the targets in the visual observations, thereby achieving accurate navigation scene perception. These methods strengthen the fusion of visual and textual information, assist the intelligent agent in capturing the key targets mentioned in the navigation instructions in complex environments, and enable the intelligent agent to make logical navigation decisions in unknown environments.

[0004] In terms of scene perception for UAV visual language navigation, since UAVs can observe reference objects from different heights and angles, this increases the difficulty of accurately identifying and understanding the same target in a complex environment. At the same time, given that the various types of targets contained in the scene are more complex in granularity, it is difficult for existing methods to achieve multi-modal alignment at a fine-grained level when mapping the entities corresponding to the phrases in the navigation instructions to the corresponding visual targets. In order to solve this key problem, UAVs need to understand the spatial relationships between various objects in a complex environment and the corresponding semantic information in the navigation instructions. This includes the determination of the relative positions between objects, the analysis of the scene layout, and the adaptation to dynamic changes. Summary of the invention

[0005] Aiming at the problem that visual language modalities are difficult to align at fine granularity due to the more complex granularity of multi-view targets, the present invention provides a UAV visual language navigation method based on a target hierarchy tree, with the goal of accurately positioning the key targets in visual observation of entities corresponding to phrases in navigation instructions.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: A UAV visual language navigation method based on a target hierarchy tree comprises the following steps: S1, obtaining the first-order logic program of the target in the navigation instruction through the target parsing module, and obtaining the text features of the navigation instruction using the text encoder; S2, construct a hierarchical tree corresponding to various targets in the front view image through the target positioning module, locate the key targets and obtain the corresponding visual features according to the first-order logic program of the target, and then use the visual encoder to extract the visual features of the front view image; S3. Use visual text features to obtain the navigation action output corresponding to each navigation moment through a multimodal encoder.

[0007] Furthermore, the target parsing module implements: A first-order logic interpreter for reference objects is constructed using a large language model to convert natural language queries into first-order logic programs, which can be used to guide the accurate identification of reference objects in subsequent scene perception. The key to this interpreter is that it can accept common language queries in the navigation field, and the generated program consists of a series of nested functions, where the output of each function can be used as the input of another function, thereby achieving a multi-level description of reference objects.

[0008] Furthermore, the target positioning module implements: By running the logic program generated by the previous interpreter through the first-order logic executor, the general logical reasoning language is executed recursively, simulating the hierarchical structure of the logical reasoning process. It can transform complex queries about reference objects in the navigation scene and accurately identify the corresponding visual targets.

[0009] Beneficial effects: 1. From the perspective of model effect, the present invention utilizes the target parsing and positioning module to greatly improve the positioning accuracy of the UAV for key targets in complex visual scenes, while enhancing the UAV's ability to understand the navigation scene, enabling it to process and parse more detailed navigation information of key targets in the scene, thereby improving the quality of navigation decisions.

[0010] 2. From a technical theory perspective, the present invention achieves a comprehensive analysis of navigation text and visual observation information through the collaborative work of the logic interpreter and the actuator. This technical approach not only improves the UAV's logical reasoning ability for navigation instructions, but also enhances its ability to make fast and accurate navigation decisions in complex environments.

[0011] 3. From the application perspective, the hierarchical tree structure design used in the present invention for target description enables the UAV navigation system to have strong scalability in the face of new navigation targets or scene changes, thereby reducing the cost of system upgrades and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of a method for visual language navigation of a drone based on a target hierarchy tree according to an embodiment of the present invention; Figure 2 Schematic diagram of a visual language navigation network model in an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0014] like Figure 1 As shown, a drone visual language navigation method based on a target hierarchy tree according to an embodiment of the present invention comprises the following steps: S1, obtaining the first-order logic program of the target in the navigation instruction through the target parsing module, and obtaining the text features of the navigation instruction using the text encoder; In this embodiment, the complexity of the UAV visual language navigation task is that it requires the UAV to perform autonomous navigation in an unknown environment according to natural language navigation instructions. This involves understanding the navigation instructions, including integrating and analyzing the actions and target information described in the instructions. In order to meet this challenge, the target parsing module uses a pre-trained large language model, takes phrases containing various types of targets in the navigation instructions as input, and finally outputs a general program represented by a logic-based reasoning language by deeply parsing and semantically understanding the navigation instructions. The program is capable of processing structured representations of multi-level descriptions of spatial and semantic targets at various levels, and this logically structured method enables the UAV to more accurately locate key targets on the navigation path when performing tasks, providing flexibility and scalability for the navigation of the UAV in complex environments, enabling it to achieve efficient and accurate navigation in a wider range of application scenarios.

[0015] In this embodiment, S1 specifically includes: S1-1: The implementation process of the target parsing module includes: taking the text description of various targets in the original navigation instruction U as input, processing it through a pre-trained large language model and generating an unambiguous first-order logic program, where the examples used to prompt the large language model are a set of general examples and system-level descriptions of first-order logic grammar rules. Each generated program consists of a function name and parameters, and can be linked together hierarchically, so that the output of one function can be used as an input parameter of another function, which is convenient for subsequent target positioning modules to process. Among them, these functions that make up the first-order logic program are predefined programs with built-in implementations, such as and, or, and not as the most basic logical functions used to connect different propositions to form more complex logical relationships. Functions such as left and behind are used to determine the orientation information between targets. Even more complex are various conceptual functions about target descriptions in navigation scenarios.

[0016] For example, the text description of the reference object is "the blue building that on front of the green building", and the first-order logic program generated by the large language model is "exist(Object, lambda x: blue(x) andbuilding(x) and ahead (x, iota(Object, lambda y: green(y) and building(x))))", where the iota function returns a vector after executing the expression and undergoing the softmax operation, and its dimension is equal to the number of targets. The entire parsing strategy maximizes the reasoning ability of the large language model while maintaining strong generalization. In addition, the entire large language model is based on the GPT-3.5 architecture, which is a powerful language model that can provide domain-independent language processing capabilities. In this way, it provides support for target detection in subsequent scene understanding.

[0017] S1-2: Use WordPieces to obtain the tokens sequence corresponding to the navigation instruction and record it as Then the sentence start identifier [CLS], navigation instruction W, and segmentation identifier [SEP] are input into the pre-trained BERT network, and finally the text features of all L+2 tokens are obtained. . Where L represents the length of the navigation instruction. Represents the i-th word in the navigation instruction.

[0018] S2. The target positioning module constructs a hierarchical tree corresponding to various targets in the front view image, locates key targets and obtains corresponding visual features according to the first-order logic program of the target, and then uses the visual encoder to extract the visual features of the front view image, including: In this embodiment, the key to the UAV visual language navigation task is to effectively combine and map the navigation instructions with the visual information in the actual environment. In order to meet this challenge, the target positioning module can deepen the understanding of complex scenes through hierarchical logical processing, so as to accurately identify the key visual targets closely related to the navigation task. First, the various targets appearing in the front view image are mainly divided into three categories through the constructed hierarchical task tree, including: 1) Key targets: reference objects that appear in navigation instructions and match navigation behaviors, which can guide the action output of the drone; 2) Target-based, reference objects that appear in navigation instructions to assist in determining the uniqueness of key targets; 3) Other targets, objects that are not in the navigation instructions but are detected in the forward-looking image. Then, the input first-order logic program is transformed effectively through the hierarchical structure of logic executors to effectively transform the complex descriptions of key targets in the navigation scene. This process greatly improves the adaptability and accuracy in processing complex navigation instructions, and provides strong support for achieving highly automated navigation decisions.

[0019] In this embodiment, S2 includes: S2-1: The implementation process of the target positioning module includes: using Faster R-CNN to detect various types of targets and extract corresponding features for the front view image obtained by the drone at the current moment. Then, the first-order logic executor is combined with the functions in the logic program to distinguish various types of targets and construct the corresponding target hierarchy tree. Among them, the corresponding multi-layer perceptron MLP is initialized for each concept function mentioned in the large language model in the target parsing module, that is, the relevant concepts in the specific field of navigation are implemented through modular neural networks to determine whether the target meets the conditions of the corresponding level. For example, given a navigation scene in which N targets are detected, the execution result of the concept function building(x) in the program can be represented as a vector of length N, and each element k in the vector corresponds to a Boolean value to indicate whether the target object k is defined as a building. Finally, the key targets in the text modality are connected with the corresponding target representations in the visual modality through the constructed hierarchical task tree, and the key target features that the drone can obtain at the navigation time t are represented as . Where n represents the number of key target features.

[0020] S2-2: The forward-looking image that the drone can receive at navigation time t is expressed as , because the visual encoder ResNet-152 has better performance on the ImageNet 2012 classification dataset than other shallow ResNet models, the visual features extracted by it have good generalization ability and are easy to migrate to other fields. Therefore, this method uses the ResNet-152 pre-trained on Place365 to extract the corresponding visual features of the received front view image .

[0021] S3. Using visual text features to obtain navigation action outputs corresponding to each navigation moment through a multimodal encoder, including: like Figure 2 As shown, for the text features obtained by the text encoder , visual features obtained by the visual encoder And the target features obtained by the target positioning module ,Will and After stacking, it is regarded as a visual modality. The navigation decision process of the drone requires a comprehensive analysis of the two modal information, text and vision. In this embodiment, considering that the lengths of different modalities may be highly unbalanced, a two-stream transformer network structure is used. This structure can balance the importance of intra-modal and inter-modal relationships through model design.

[0022] For each layer of the multimodal encoder, the relationship between modalities is first learned through modal interaction, that is, visual-text cross-attention is performed on the visual modality to focus on the relevant textual information. Similarly, visual-text cross-attention is performed on the textual modality to focus on the relevant visual information. Then the relationship within the text and visual modalities is learned through modal self-attention. Finally, the multimodal encoder outputs the updated text features and visual features , and use the fully connected layer to transform the visual features Mapped to the action output at the corresponding moment.

[0023] The loss function used in model training As shown below: ; Among them, for action prediction, cross entropy minimization is used to predict navigation actions and truth value navigation actions The loss between.

[0024] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A UAV visual language navigation method based on a target hierarchy tree, characterized in that: The steps include: S1, obtaining the first-order logic program of the target in the navigation instruction through the target parsing module, and obtaining the text features of the navigation instruction using the text encoder; S2, construct a hierarchical tree corresponding to various targets in the front view image through the target positioning module, locate the key targets and obtain the corresponding visual features according to the first-order logic program of the target, and then use the visual encoder to extract the visual features of the front view image; S3. Use visual text features to obtain the navigation action output corresponding to each navigation moment through a multimodal encoder.

2. The method for unmanned aerial vehicle visual language navigation based on target hierarchy tree according to claim 1 is characterized in that: In S1, the target parsing module implements: A first-order logic interpreter for reference objects is constructed using a large language model to convert natural language queries into first-order logic programs. The first-order logic programs are used to guide the accurate identification of reference objects in subsequent scene perception. The first-order logic interpreter accepts language queries commonly used in the navigation field, and the generated program consists of a series of nested functions, in which the output of each function serves as the input of another function, thereby realizing a multi-level description of reference objects.

3. The method for unmanned aerial vehicle visual language navigation based on target hierarchy tree according to claim 1 is characterized in that: In S2, the target positioning module implements: The first-order logic program generated by the first-order logic interpreter is run through the first-order logic executor, and the general logical reasoning language is executed recursively, simulating the hierarchical structure of the logical reasoning process, transforming complex queries about reference objects in the navigation scene and accurately identifying the corresponding visual targets.

Images