Unmanned aerial vehicle visual angle three-dimensional visual positioning method

Abstract

The invention discloses an unmanned aerial vehicle visual angle three-dimensional visual positioning method, which comprises the following steps: extracting candidate objects and landmarks from a three-dimensional scene, extracting object features by using an object encoder, and extracting landmark features by using a landmark encoder; constructing a language analysis strategy based on prompt learning, and analyzing the overall natural language description into a plurality of partial language descriptions; constructing an object feature enhancement module based on landmark reference, describing and mining neighbor features between an object and a landmark by using a part of natural languages to enhance feature representation of objects near the landmark, and guiding a positioning network to pay attention to the objects near the landmark; an object fine positioning module based on relation perception is designed to make full use of relation information between objects, and unmanned aerial vehicle visual angle three-dimensional visual positioning is achieved.

Description

Technical Field

[0001] This invention relates to the fields of three-dimensional visual positioning and computer vision, and in particular to a three-dimensional visual positioning method from the perspective of an unmanned aerial vehicle (UAV). Background Technology

[0002] 3D visual positioning aims to locate target objects of interest in a 3D scene based on natural language descriptions. With the rapid development of artificial intelligence and computer vision technologies, 3D visual positioning has made significant progress. Currently, with the increasing prevalence of applications such as low-altitude economics and aerial search and rescue, UAV-based 3D visual positioning for urban scenarios is receiving growing attention. However, UAV-perspective scenes typically cover a wide area and often include various elements such as buildings, roads, and vehicles. This makes accurate matching between visual targets in the 3D scene and natural language descriptions difficult, limiting the accuracy of 3D visual positioning. Therefore, achieving effective and accurate UAV-perspective 3D visual positioning remains a challenge.

[0003] In recent years, significant progress has been made in 3D visual localization from the perspective of unmanned aerial vehicles (UAVs). Miyanishi et al. achieved target object localization by fusing linguistic features, target object features, and landmark features, and then converting the fused features into confidence scores. Li et al. proposed a two-stage 3D visual localization method, which first filters out language-related candidate objects, then calculates the similarity between candidate object features and linguistic features, and finally determines the target object. In the visual localization process, the above methods typically extract sentence-level linguistic features to achieve matching between natural language descriptions and visual targets.

[0004] However, to effectively locate target objects in a large-scale scene from a UAV's perspective, the natural language description used for visual localization typically contains rich multidimensional information, such as proximity information between the target object and landmarks, and spatial relationship information between objects. Simply extracting sentence-level linguistic features is usually insufficient to fully exploit the rich multidimensional information in the natural language description, thus limiting the accuracy of 3D visual localization from a UAV's perspective. Summary of the Invention

[0005] This invention provides a UAV-based 3D visual positioning method. It utilizes a large language model to parse nearest neighbor information and spatial relationship information between objects and landmarks from natural language descriptions. Guided by the obtained multi-dimensional information, a coarse-to-fine strategy is employed to guide the positioning process, thereby achieving effective UAV-based 3D visual positioning. Details are described below: A three-dimensional visual positioning method from the perspective of an unmanned aerial vehicle (UAV), the method comprising: Extract candidate objects and landmarks from a 3D scene, extract object features using an object encoder, and extract landmark features using a landmark encoder; Construct a language parsing strategy based on prompting learning to parse the overall natural language description into multiple part language descriptions; A landmark-based object feature enhancement module is constructed, which utilizes some natural language descriptions to mine the proximity characteristics between objects and landmarks, thereby enhancing the feature representation of objects near landmarks and guiding the localization network to focus on objects near landmarks. We designed a relationship-aware object fine localization module to fully utilize the information about the relationships between objects and achieve 3D visual localization from the perspective of a UAV.

[0006] The method further includes: training a UAV-view 3D visual positioning network, and locating 3D visual targets based on the trained positioning network.

[0007] The proposed language parsing strategy based on prompting learning parses the overall natural language description into multiple partial language descriptions: Construct language parsing hints, the hints are generated by It consists of several language instances, each containing the original natural language description and several parsed partial language descriptions, namely the target object attribute language description. Nearby information language description Reference object attribute language description and relational information language description ; The language parsing prompts and the natural language description to be processed are input into a pre-trained large language model. The language parsing prompts are used to guide the large language model to learn language parsing rules, so as to accurately parse the natural language description to be processed into multiple parts of language description.

[0008] The object feature enhancement module uses target features Landmark features Target object attribute language description and neighbor information language description Using this as input, we ultimately obtain object features from cross-modal nearest neighbor perception. .

[0009] Among them, the object features obtained through cross-modal nearest neighbor perception are described. for: Design nearest neighbor feature mining units to focus on those that match the nearest neighbor information description. The potential target object is determined by calculating the nearest neighbor weights controlled by language. Evaluate each object and its neighbor information in a language description The closer an object is to a landmark, the higher its weight is assigned, while objects that are farther away receive a lower weight. Enhanced object features using target attributes Computational query embedding Based on landmark features Calculate the corresponding key embeddings respectively Sum embedding Based on query embedding and key embedding Calculate the attention matrix between the target object and the landmark. To learn the relationship between objects and landmarks; Using language-controlled nearest neighbor weights Adjusting the attention matrix between the target object and the landmark Generate the nearest neighbor attention matrix Integrate neighboring information into the attention matrix; Calculate object features perceived by nearest neighbors using an attention mechanism. Through a cross-attention mechanism, the object features obtained from nearest neighbor perception are calculated. Language features of neighboring information Fusion to obtain object features from cross-modal nearest neighbor perception. .

[0010] The object fine localization module uses cross-modal nearest neighbor perception of object features. Reference object attribute language description and relational information language description As input, obtain object features for cross-modal relationship perception. .

[0011] Furthermore, the object features obtained for cross-modal relationship perception for: Obtain the language features of the reference object's attributes Object features perceived across modal nearest neighbors and Perform fusion to obtain object features with enhanced reference object properties. Design spatial relationship mining units to locate elements that conform to relational information language descriptions. For the target object, in the spatial relationship mining unit, the relational weights controlled by language are learned. ; Enhanced object features using reference object properties Computational query embedding Key embedding and value embedding Based on query embedding and key embedding Calculate the attention matrix between objects ; Weighting relationships controlled by language Inter-object attention matrix Fusion to obtain the relational attention matrix In the relational attention matrix Guided by this, computational relationship-aware object features Object features perceived through fusion relationships Corresponding relational language features To obtain object features for cross-modal relation perception. .

[0012] The beneficial effects of the technical solution provided by this invention are: 1. This invention utilizes a large language model to deeply mine the rich multidimensional information in natural language descriptions, effectively alleviating the problem of insufficient utilization of information in natural language descriptions by existing methods, and improving the performance of UAV perspective three-dimensional visual positioning. 2. This invention constructs an object feature enhancement module based on landmark reference. This module fully exploits the proximity characteristics between the target object and the landmark, enabling the localization network to focus on objects near the landmark, thereby reducing interference from irrelevant objects. 3. The present invention designs a relationship-aware object fine positioning module. This module further locates the position of the target object by utilizing the spatial relationship between the target object and the reference object, thereby achieving effective three-dimensional visual positioning. Attached Figure Description

[0013] Figure 1 A flowchart for a 3D visual positioning method from the perspective of an unmanned aerial vehicle (UAV). Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0015] This invention provides a three-dimensional visual positioning method from the perspective of an unmanned aerial vehicle (UAV), see [link / reference]. Figure 1 The method includes the following steps: I. Constructing an object and landmark feature encoder This method takes a 3D scene and natural language description as input, and first extracts candidate objects and landmarks from the 3D scene. Then, it uses an object encoder to extract object features. At the same time, landmark encoders are used to extract landmark features. Both the object encoder and the landmark encoder are constructed using sparse convolutional networks. Sparse convolutional networks are well known to those skilled in the art, and will not be described in detail in this embodiment of the invention.

[0016] II. Designing a Language Parsing Strategy Based on Prompt Learning Natural language description in UAV-based 3D visual localization tasks Natural language descriptions typically contain rich, multidimensional information. This invention employs a prompt-based learning-based language parsing strategy to parse the overall natural language description into multiple partial language descriptions, thereby fully utilizing the rich, multidimensional information within the natural language description.

[0017] Specifically, firstly, language parsing hints are constructed, and the hints are generated by... The system consists of several language instances, each containing the original natural language description and multiple parsed partial language descriptions. Specifically, the parsed partial language descriptions include target object attribute descriptions, nearest neighbor information descriptions, reference object attribute descriptions, and relational information descriptions. Subsequently, the language parsing hints and the natural language description to be processed are input into a pre-trained large language model. The language parsing hints guide the large language model to learn language parsing rules, thereby accurately parsing the natural language description to be processed into multiple partial language descriptions. The formula for the parsing process is as follows:

[0018] in, This indicates a language parsing hint. Represents a large language model. , , and These represent the parsed target object attribute language description, nearest neighbor information language description, reference object attribute language description, and relation information language description, respectively. In this embodiment of the invention, the language model used is the Tongyi Qianwen 2.5. The value is set to 10.

[0019] III. Constructing an object feature enhancement module based on landmark reference Considering that target objects are usually located near landmarks, this embodiment of the invention constructs a landmark-based object feature enhancement module. It utilizes natural language descriptions to mine the proximity characteristics between objects and landmarks, thereby enhancing the feature representation of objects near landmarks. This guides the localization network to focus more on objects near landmarks and reduces interference from irrelevant objects. The designed landmark-based object feature enhancement module uses target features... Landmark features Target object attribute language description and neighbor information language description Using this as input, we ultimately obtain object features from cross-modal nearest neighbor perception. .

[0020] Specifically, the parsed target object attribute language description Nearby information language description The input is fed into a text encoder to obtain the linguistic features of the target object's attributes. and neighbor information language features The text encoding process is represented as follows:

[0021]

[0022] in, This indicates a text encoder.

[0023] Obtain the linguistic features of the target object's attributes Then, by fusing target features and Establish an awareness of target attributes to obtain object features with enhanced target attributes. Then, a nearest neighbor feature mining unit was designed to focus on features that match the nearest neighbor information description. Potential target objects. In the nearest neighbor feature mining unit, the nearest neighbor weights are calculated using language control. Evaluate each object and its neighbor information in a language description The matching degree, that is, objects closer to the landmark are given higher weights, while objects farther away receive lower weights. Nearest neighbor weights. The calculation formula is as follows:

[0024] in, Indicates feature splicing, These are learnable parameters. This represents the distance between the target object and the landmark, calculated using the Euclidean distance between the center of the landmark and the center of the object. Subsequently, object features are enhanced based on target attributes. Calculate query embedding while utilizing landmark features The corresponding key-value embeddings are calculated separately, as shown in the following formulas:

[0025] in, These are learnable parameters. It is by The query embedding obtained from the mapping, It is by The key embedding obtained from the mapping, It is by The values ​​obtained from the mapping are embedded.

[0026] Based on query embedding and key embedding Calculate the attention matrix between the target object and the landmark. The formula for learning the relationship between objects and landmarks is as follows:

[0027] in, dim This represents the dimension of query and key embedding. In this embodiment of the invention, dim The value is set to 128.

[0028] Then, the language-controlled nearest neighbor weights are utilized. Adjusting the attention matrix between the target object and the landmark Generate the nearest neighbor attention matrix This allows for the integration of neighboring information into the attention matrix, calculated using the following formula:

[0029] in, This indicates exponentiation. and Both represent activation functions.

[0030] Finally, the object features perceived by nearest neighbors are calculated through an attention mechanism. The formula for the above process is expressed as follows:

[0031] Finally, through a cross-attention mechanism, the calculated nearest-neighbor perception object features are... Language features of neighboring information Fusion, thereby obtaining object features from cross-modal nearest neighbor perception. .

[0032] IV. Design of a Relationship-Aware Fine Object Localization Module Typically, each object exhibits a specific spatial relationship with its surrounding objects. Thoroughly exploring these spatial relationships helps determine the precise location of a target object. Therefore, this invention designs a relationship-aware object fine-tuning module to fully utilize the information about the relationships between objects, thereby achieving effective UAV-based 3D visual localization. The designed relationship-aware object fine-tuning module uses cross-modal nearest neighbor perception of object features. Reference object property language description and relational information language description As input, obtain object features for cross-modal relationship perception. .

[0033] Specifically, the parsed reference object attribute language description is first... Relationship information language description Input the text encoder separately to obtain the language features of the reference object attributes. Relational information language features The text encoding process is represented as follows:

[0034]

[0035] Obtain the language features of the reference object's attributes Then, the object features perceived across modal nearest neighbors will be... and Perform fusion to obtain object features with enhanced reference object properties. Then, a spatial relationship mining unit was designed to locate elements that conform to the relational information language description. The target object. In the spatial relationship mining unit, the relation weights controlled by language are learned first. The formula is as follows:

[0036] in, These are learnable parameters. This indicates feature splicing. Represents the offset matrix. and They represent the first From the center of the first object to the... The offset of the center of an object from the bird's-eye view.

[0037] Subsequently, the object features are enhanced using the properties of the reference object. The formulas for calculating query embedding, key embedding, and value embedding are as follows:

[0038] in These are learnable parameters. Indicates by the characteristics of the object The resulting queries, key, and value embeddings are obtained from the mapping.

[0039] Then, by calculating the query embedding and key embedding Attention matrix To establish connections between objects, the formula is as follows:

[0040] in, This represents the attention matrix between objects. In this embodiment of the invention... dim The value is set to 128.

[0041] To further effectively perceive the relationships between objects that match natural language descriptions, the language-controlled relationship weights will be adjusted. and Fusion to obtain the relational attention matrix The calculation formula is as follows:

[0042] Subsequently, in the relational attention matrix Guided by this, computational relationship-aware object features This allows for the effective perception of relationships between objects that match natural language descriptions. The specific process is as follows:

[0043] Then, object features perceived through fusion relationships. Corresponding relational language features This enables effective interaction between cross-modal features, resulting in object features with cross-modal relationship awareness. Finally, the target object localization head perceives object features through cross-modal relationships. Using this as input, 3D visual localization is performed. Specifically, a classifier is used to output the probability that each candidate object is the target object. and in Select the object with the highest probability as the target object for prediction. The formula for the above process is expressed as follows:

[0044]

[0045] in, This indicates the category header. This represents the probability that each candidate object is the target object. This indicates the number of candidate objects.

[0046] V. Training a 3D Visual Positioning Network from the Perspective of an Unmanned Aerial Vehicle The UAV-based 3D visual localization network proposed in this embodiment is trained in an end-to-end manner. During training, a loss function is introduced to constrain network training. First, a sentence classification loss is introduced. This ensures the quality of the language features output by the text encoder. Specifically, the text encoder is used to predict the category of the target object corresponding to the natural language description. Then, use statement classification loss. Measure the predicted target object category and the actual target object category The differences between them.

[0047] Subsequently, visual localization loss was introduced. Target objects predicted by computer vision localization networks and real target object The differences between them. Finally, the constructed loss function formula is expressed as follows:

[0048] in, This represents the introduced statement classification loss. Indicates visual localization loss. , These represent the weight coefficients of the sentence classification loss and the visual localization loss, respectively. In this embodiment of the invention, the cross-entropy loss function is used as both the sentence classification loss and the visual localization loss. Set to 0.5. Set to 10. The cross-entropy loss function is well-known in the field and will not be elaborated upon here.

[0049] After training the localization network, the designed 3D visual localization network first loads the weights of the pre-trained model, then inputs the natural language descriptions of the 3D scene and the target object into the network, and finally outputs the predicted target object, thereby determining the position of the target object in the 3D scene and ultimately achieving 3D visual localization. Furthermore, this model can be applied to fields such as UAV transportation and disaster aerial search and rescue, providing reliable support for UAVs' autonomous perception and precise operations in 3D environments.

[0050] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0051] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0052] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A three-dimensional visual positioning method from the perspective of an unmanned aerial vehicle (UAV), characterized in that, The method includes: Extract candidate objects and landmarks from a 3D scene, extract object features using an object encoder, and extract landmark features using a landmark encoder; Construct a language parsing strategy based on prompting learning to parse the overall natural language description into multiple part language descriptions; A landmark-based object feature enhancement module is constructed, which utilizes some natural language descriptions to mine the proximity characteristics between objects and landmarks, thereby enhancing the feature representation of objects near landmarks and guiding the localization network to focus on objects near landmarks. We designed a relationship-aware object fine localization module to fully utilize the information about the relationships between objects and achieve 3D visual localization from the perspective of a UAV.

2. The UAV-based three-dimensional visual positioning method according to claim 1, characterized in that, The method further includes: training a UAV-view 3D visual positioning network, and locating 3D visual targets based on the trained positioning network.

3. The UAV-based three-dimensional visual positioning method according to claim 1, characterized in that, The constructed prompt-based language parsing strategy parses the overall natural language description into multiple participial language descriptions: Build language parsing hints The prompt is from It consists of several language instances, each containing the original natural language description and several parsed partial language descriptions, namely the target object attribute language description. Nearby information language description Reference object attribute language description and relational information language description ; Language parsing prompts and the natural language description to be processed The input is fed into a pre-trained large language model, and language parsing prompts are used to guide the large language model to learn language parsing rules, so as to accurately parse the natural language description to be processed into multiple part language descriptions.

4. The UAV-based three-dimensional visual positioning method according to claim 1, characterized in that, The object feature enhancement module uses target features Landmark features Target object attribute language description and neighbor information language description Using this as input, we ultimately obtain object features from cross-modal nearest neighbor perception. .

5. A UAV-based three-dimensional visual positioning method according to claim 4, characterized in that, The object features obtained through cross-modal nearest neighbor perception. for: Design nearest neighbor feature mining units to focus on those that match the nearest neighbor information description. The potential target objects are evaluated by computing language-controlled nearest neighbor weights to assess each object's relationship with its nearest neighbor information in a linguistic description. The closer an object is to a landmark, the higher its weight is assigned, while objects that are farther away receive a lower weight. Enhanced object features using target attributes Computational query embedding Based on landmark features Calculate the corresponding key embeddings respectively Sum embedding ; Based on query embedding and key embedding Calculate the attention matrix between the target object and the landmark. To learn the relationship between objects and landmarks; Using language-controlled nearest neighbor weights Adjusting the attention matrix between the target object and the landmark Generate the nearest neighbor attention matrix Integrate neighboring information into the attention matrix; Calculate object features perceived by nearest neighbors using an attention mechanism. Through a cross-attention mechanism, the object features obtained from nearest neighbor perception are calculated. Language features of neighboring information Fusion to obtain object features from cross-modal nearest neighbor perception. .

6. The UAV-based three-dimensional visual positioning method according to claim 1, characterized in that, The object fine localization module uses cross-modal nearest neighbor perception of object features. Reference object attribute language description and relational information language description As input, obtain object features for cross-modal relationship perception. .

7. A UAV-based three-dimensional visual positioning method according to claim 6, characterized in that, The object features obtained for cross-modal relation perception for: Obtain the language features of the reference object's attributes Object features perceived across modal nearest neighbors Language features of reference object attributes Perform fusion to obtain object features with enhanced reference object properties. Design spatial relationship mining units to locate elements that conform to relational information language descriptions. For the target object, in the spatial relationship mining unit, the relational weights controlled by language are learned. ; Enhanced object features using reference object properties Computational query embedding Key embedding and value embedding Based on query embedding and key embedding Calculate the attention matrix between objects ; Weighting relationships controlled by language Inter-object attention matrix Fusion to obtain the relational attention matrix In the relational attention matrix Guided by this, computational relationship-aware object features Object features perceived through fusion relationships Corresponding relational language features To obtain object features for cross-modal relation perception. .

Images