A method and device for unmanned aerial vehicle visual language navigation based on geometric information enhancement

Abstract

This application provides a method and apparatus for UAV visual language navigation based on geometric information enhancement. The method includes: performing bi-branch feature extraction on multi-view RGB images to obtain two-dimensional first visual semantic features and three-dimensional implicit geometric features; introducing the overall structural cues of the implicit geometric features into the first visual semantic features using a geometric prior injection model to obtain second visual semantic features; aligning and fusing the second visual semantic features and implicit geometric features using a geometric perception reparameter model to obtain an enhanced visual representation; fusing the enhanced visual representation, task language commands, and UAV status using a large language model to obtain the current time-to-time trajectory increment; and determining the predicted trajectory point of the UAV at the next time-to-time based on the current UAV trajectory point and trajectory increment. This application improves the spatial understanding ability and navigation trajectory prediction stability of UAVs in complex environments.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation technology for unmanned aerial vehicles (UAVs), and specifically to a visual language navigation method and apparatus for UAVs based on geometric information enhancement. Background Technology

[0002] With the development of artificial intelligence, computer vision, and natural language processing technologies, visual language navigation methods have gradually become an important direction in UAV autonomous navigation research. UAV visual language navigation tasks require UAVs to autonomously generate flight trajectories and complete navigation tasks in complex three-dimensional environments based on natural language commands and visual observations.

[0003] Unlike terrestrial robot navigation, drones typically rely on multi-view 2D RGB images to perceive their environment, but their navigation decisions are made in a continuous 3D space. Therefore, there is an inherent difference between 2D visual observation and 3D trajectory decision-making. Without effective spatial representation methods, models often struggle to stably understand scene structure and maintain consistency across different viewpoints, leading to unstable 3D path predictions.

[0004] Most existing UAV visual-language navigation methods directly predict paths based on current images and language commands. Although some methods have improved navigation performance through discrete action space, explicit geometric modeling, or large language models, these methods are mostly still at the policy level and have not truly solved the problem of representation connection between two-dimensional visual perception and three-dimensional spatial decision-making.

[0005] In addition, some methods enhance spatial awareness through depth recovery, bird's-eye view, semantic map or explicit 3D modeling, but these methods usually rely on additional geometric modules, which increases system complexity and is not conducive to lightweight deployment. Summary of the Invention

[0006] In view of this, this application provides a method and apparatus for UAV visual language navigation based on geometric information enhancement to solve the above-mentioned technical problems.

[0007] In a first aspect, embodiments of this application provide a UAV visual language navigation method based on geometric information enhancement, comprising: Acquire multi-view RGB images of the target area at the current moment, task language commands, and drone status; A two-branch feature extraction method is used to extract RGB images from multiple perspectives to obtain two-dimensional first visual semantic features and three-dimensional implicit geometric features. By using a pre-trained geometric prior injection model, the overall structural cues of implicit geometric features are introduced into the first visual semantic features to obtain the second visual semantic features. The second visual semantic features and implicit geometric features are aligned and fused using a pre-trained geometric perception reparameter model to obtain an enhanced visual representation; By using a pre-trained large language model to fuse the enhanced visual representation, task language commands, and UAV status, the current trajectory increment is obtained. Based on the current drone trajectory points and track increments, the predicted drone trajectory points for the next moment are determined.

[0008] In one possible implementation, the current moment Multi-view RGB images Represented as: in, , , , and RGB images representing the front view, rear view, left view, right view, and bottom view, respectively; Current moment drone status Represented as a six-degree-of-freedom pose : in, This indicates the position coordinates of the drone in three-dimensional space. These represent the roll angle, pitch angle, and yaw angle, respectively.

[0009] In one possible implementation, the dual branches include a parallel Clip 2D visual encoder and a VGGT geometric encoder; A two-branch feature extraction method is used to extract features from multi-view RGB images, resulting in two-dimensional first visual semantic features and three-dimensional implicit geometric features; including: Using the Clip 2D visual encoder to process RGB images from multiple perspectives Encoding is performed to obtain the first two-dimensional visual semantic features: , in, Indicates Clip 2D visual encoder; Indicates the number of two-dimensional visual tokens. This represents the feature dimension output by the two-dimensional visual encoder; Using the VGGT geometric encoder to process multi-view RGB images Encoding is performed to obtain three-dimensional implicit geometric features. : , in, This indicates the VGGT geometry encoder. Indicates the number of 3D geometry tokens. This represents the feature dimension output by the VGGT geometry encoder.

[0010] In one possible implementation, the geometric prior injection model includes: a first branch, a second branch, and a computation unit, wherein the first branch and the second branch are parallel, and the computation unit is connected to both the first branch and the second branch. The first branch includes a first linear projection layer; the second branch includes: a second linear projection layer, an average pooling layer, two multilayer perceptrons, and a sigmoid function connected in sequence. By utilizing a pre-trained geometric prior injection model, the overall structural cues of implicit geometric features are introduced into the first visual semantic feature, resulting in the second visual semantic feature; including: The basic query representation is obtained by linearly projecting the first visual semantic features using the first linear projection layer. : in, For learnable projection matrices, For the unified feature dimensions; Using the second linear projection layer for implicit geometric features Perform linear projection to obtain features : in, For learnable projection matrices, For bias terms; Using average pooling layers for features Perform mean pooling to obtain the global geometric summary vector. ; Using two multilayer perceptrons to summarize the global geometric vector Modulation processing is performed to obtain the first scaling parameter used to modulate the two-dimensional features. and offset parameters : in, It is a two-layer, multi-layer sensor. Indicate its parameters; Using the Sigmoid function to scale the first parameter The second scaling parameter is obtained through processing. : in, Represents the Sigmoid function; Representing the basic query using computational units Second scaling parameter and offset parameters The process is performed to obtain the second visual semantic features enhanced by geometric prior. : in, This represents the learnable injection strength coefficient. This indicates element-wise multiplication.

[0011] In one possible implementation, the geometry-aware reparameter model includes: a third linear mapping layer, a fourth linear mapping layer, a multi-head attention unit, a stitching unit, a fifth linear mapping layer, and a fusion unit. The third and fourth linear mapping layers are parallel, the multi-head attention unit is connected to both the third and fourth linear mapping layers, and the multi-head attention unit, stitching unit, fifth linear mapping layer, and fusion unit are connected sequentially. An enhanced visual representation is obtained by aligning and fusing second visual semantic features and implicit geometric features using a pre-trained geometric perception reparameter model; including: Using the third linear mapping layer for implicit geometric features Perform a linear mapping to obtain key features. : in, Projection matrix, For bias terms; Using the fourth linear mapping layer for implicit geometric features Perform a linear mapping to obtain value characteristics. : in, Projection matrix, For bias terms; Using inclusion A multi-head attention unit focuses on second visual semantic features and key features. and value characteristics Processing is performed to obtain Each attention output: ; Using splicing units for all The attention outputs are concatenated to obtain the concatenated features; The fifth linear mapping layer is used to perform linear mapping on the concatenated features to obtain the aligned visual representation. ; Using fusion units to align the visual representation Compared with the basic query representation By fusing the data, an enhanced visual representation is obtained. : in, For learnable gating coefficients, .

[0012] In one possible implementation, a pre-trained large language model is used to fuse the enhanced visual representation, task language commands, and UAV state information to obtain the current time-to-time trajectory increment, including: Using large language models for language instructions Enhance visual representation and drone status Processing is performed to obtain the hidden representation. ; in, This represents the function for constructing multimodal prompt words. Represents the processing functions of a large language model; Using track regression head pairs to hide representations Processing is performed to obtain the track increment. : in, This indicates the return trajectory.

[0013] In one possible implementation, the method further includes: The steps for jointly training the geometric prior injection model, the geometrically aware reparameterized model, the large language model, and the track regression head.

[0014] Secondly, embodiments of this application provide a UAV visual language navigation device based on geometric information enhancement, comprising: The acquisition unit is used to acquire multi-view RGB images of the target area at the current moment, task language commands, and UAV status; The feature extraction unit is used to perform bi-branch feature extraction on RGB images from multiple perspectives to obtain two-dimensional first visual semantic features and three-dimensional implicit geometric features. The first processing unit is used to introduce the overall structural cues of implicit geometric features into the first visual semantic features using a pre-trained geometric prior injection model, thereby obtaining the second visual semantic features. The second processing unit is used to align and fuse the second visual semantic features and implicit geometric features using a pre-trained geometric perception reparameter model to obtain an enhanced visual representation. The inference unit is used to fuse the enhanced visual representation, task language commands, and UAV status using a pre-trained large language model to obtain the current trajectory increment. The determination unit is used to determine the predicted trajectory point of the UAV at the next moment based on the UAV trajectory point and trajectory increment at the current moment.

[0015] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the methods of embodiments of this application.

[0017] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the method of embodiments of this application.

[0018] This application improves the spatial understanding and navigation trajectory prediction stability of UAVs in complex environments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating the UAV visual-language navigation method based on geometric information enhancement provided in this application embodiment; Figure 2 A functional structure diagram of a UAV visual language navigation device based on geometric information enhancement provided in an embodiment of this application; Figure 3 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0023] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0024] To address the issues of mismatch between 2D visual perception and 3D continuous trajectory decision-making, insufficient spatial consistency, and unstable trajectory prediction in complex environments in existing UAV visual language navigation methods, this application provides a UAV visual language navigation method based on geometric information enhancement. The method includes: extracting 2D visual semantic features and implicit 3D geometric features from multi-view RGB image observations acquired by the UAV at the current moment; then, using geometric prior injection, introducing the overall structural cues from the implicit 3D geometric information into the 2D semantic features; further aligning and fusing the 2D semantic features and 3D geometric features through geometric perception reparameterization to obtain an enhanced visual representation that combines semantic and spatial structural information; finally, inputting the enhanced visual representation, task language commands, and UAV state information into the navigation decision module to output the UAV's next trajectory point, trajectory increment, or control result.

[0025] This application departs from directly predicting 3D trajectories from 2D image features. Instead, it first supplements the feature level with geometric constraints, making the visual representation more suitable for subsequent 3D spatial navigation decisions. This approach alleviates the representation mismatch problem between 2D perception and 3D decision-making, improves the UAV's understanding of complex scene structures, and enhances the continuity, stability, and reliability of the navigation trajectory. Simulation visualization analysis also demonstrates that this method enables paths to more closely approximate real trajectories, resulting in smoother and more stable motion, while simultaneously making feature responses more structured and regularized.

[0026] This application has the following beneficial effects: 1. Implicit geometric information that is helpful for navigation can be extracted using only multi-view RGB images, without the need for explicit 3D reconstruction, depth restoration or point cloud generation, thus reducing system complexity.

[0027] 2. By injecting geometric priors and reparameterizing geometric perception, two-dimensional visual features can carry stronger spatial structure information, thereby alleviating the representation mismatch problem between two-dimensional visual perception and three-dimensional trajectory decision-making.

[0028] 3. It can improve the spatial understanding ability of UAVs in complex and unknown environments, making the navigation trajectory smoother and more stable, and improving the overall navigation reliability.

[0029] This application is applicable to autonomous navigation missions of unmanned aerial vehicles (UAVs) in complex environments such as post-disaster search and rescue, urban inspection, autonomous exploration, and infrastructure inspection.

[0030] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0031] like Figure 1 As shown, this application provides a UAV visual language navigation method based on geometric information enhancement, including: Step 101: Acquire multi-view RGB images of the target area at the current moment, task language commands, and UAV status; Step 102: Perform bi-branch feature extraction on the multi-view RGB images to obtain two-dimensional first visual semantic features and three-dimensional implicit geometric features; Step 103: Using a pre-trained geometric prior injection model, the overall structural cues of the implicit geometric features are introduced into the first visual semantic features to obtain the second visual semantic features. Step 104: Align and fuse the second visual semantic features and implicit geometric features using the pre-trained geometric perception reparameter model to obtain the enhanced visual representation; Step 105: Use the pre-trained large language model to fuse the enhanced visual representation, task language commands, and UAV status to obtain the current trajectory increment; Step 106: Based on the current UAV trajectory points and track increments, determine the predicted UAV trajectory points for the next moment.

[0032] This embodiment extracts two-dimensional visual semantic features and implicit three-dimensional geometric features from the same set of multi-view RGB images. Then, it uses a geometric prior injection model and a geometric perception reparameterization model to enhance and align the two-dimensional semantic features. Finally, the enhanced visual representation, along with language commands and UAV status, is input into the navigation decision model to output the next trajectory increment. This improves the UAV's spatial understanding ability and navigation trajectory prediction stability in complex environments.

[0033] In some embodiments, the current time Multi-view RGB images Represented as: in, , , , and RGB images representing the front view, rear view, left view, right view, and bottom view, respectively; Current moment drone status Represented as a six-degree-of-freedom pose : in, This indicates the position coordinates of the drone in three-dimensional space. These represent the roll angle, pitch angle, and yaw angle, respectively.

[0034] In some embodiments, the dual branches include a parallel Clip 2D visual encoder and a VGGT geometric encoder; A two-branch feature extraction method is used to extract features from multi-view RGB images, resulting in two-dimensional first visual semantic features and three-dimensional implicit geometric features; including: Using the Clip 2D visual encoder to process RGB images from multiple perspectives Encoding is performed to obtain the first two-dimensional visual semantic features: , in, Indicates Clip 2D visual encoder; Indicates the number of two-dimensional visual tokens. This represents the feature dimension output by the two-dimensional visual encoder; Using the VGGT geometric encoder to process multi-view RGB images Encoding is performed to obtain three-dimensional implicit geometric features. : , in, This indicates the VGGT geometry encoder. Indicates the number of 3D geometry tokens. This represents the feature dimension output by the VGGT geometry encoder.

[0035] In some embodiments, the geometric prior injection model includes: a first branch, a second branch, and a computation unit, wherein the first branch and the second branch are parallel, and the computation unit is connected to both the first branch and the second branch. The first branch includes a first linear projection layer; the second branch includes: a second linear projection layer, an average pooling layer, two multilayer perceptrons, and a sigmoid function connected in sequence. By utilizing a pre-trained geometric prior injection model, the overall structural cues of implicit geometric features are introduced into the first visual semantic feature, resulting in the second visual semantic feature; including: To map the two-dimensional features to the shared feature space, a first linear projection layer is used to linearly project the first visual semantic features to obtain the basic query representation. : in, For learnable projection matrices, For the unified feature dimensions; To place geometric features and two-dimensional features in the same dimensional space, a second linear projection layer is used to apply implicit geometric features. Perform linear projection to obtain features : in, For learnable projection matrices, For bias terms; Using average pooling layers for features Perform mean pooling to obtain the global geometric summary vector. ; Using two multilayer perceptrons to summarize the global geometric vector Modulation processing is performed to obtain the first scaling parameter used to modulate the two-dimensional features. and offset parameters : in, It is a two-layer, multi-layer sensor. Indicate its parameters; To improve stability, the Sigmoid function is used to adjust the first scaling parameter. The second scaling parameter is obtained through processing. : in, Represents the Sigmoid function; Representing the basic query using computational units Second scaling parameter and offset parameters The process is performed to obtain the second visual semantic features enhanced by geometric prior. : in, This represents the learnable injection strength coefficient. This indicates element-wise multiplication.

[0036] The geometric prior injection model in this embodiment can inject global spatial structure information from implicit three-dimensional geometric features into two-dimensional visual semantic features, enabling the two-dimensional semantic features to retain their original semantic discrimination capabilities while possessing scene-level geometric perception capabilities.

[0037] In some embodiments, the geometric perception reparameter model includes: a third linear mapping layer, a fourth linear mapping layer, a multi-head attention unit, a splicing unit, a fifth linear mapping layer, and a fusion unit, wherein the third linear mapping layer and the fourth linear mapping layer are parallel, the multi-head attention unit is connected to both the third linear mapping layer and the fourth linear mapping layer, and the multi-head attention unit, the splicing unit, the fifth linear mapping layer, and the fusion unit are connected sequentially. An enhanced visual representation is obtained by aligning and fusing second visual semantic features and implicit geometric features using a pre-trained geometric perception reparameter model; including: Using the third linear mapping layer for implicit geometric features Perform a linear mapping to obtain key features. : in, Projection matrix, For bias terms; Using the fourth linear mapping layer for implicit geometric features Perform a linear mapping to obtain value characteristics. : in, Projection matrix, For bias terms; Using inclusion A multi-head attention unit focuses on second visual semantic features and key features. and value characteristics Processing is performed to obtain Each attention output: ; Using splicing units for all The attention outputs are concatenated to obtain the concatenated features; The fifth linear mapping layer is used to perform linear mapping on the concatenated features to obtain the aligned visual representation. ; Using fusion units to align the visual representation Compared with the basic query representation By fusing the data, an enhanced visual representation is obtained. : in, For learnable gating coefficients, .

[0038] Through the above process, two-dimensional semantic features can selectively absorb more useful structural information from geometric features based on their own content. Then, through gated residual fusion, the original semantic information is preserved, ultimately forming an enhanced visual representation that is more suitable for three-dimensional navigation decisions. The geometry-aware reparameterization model in this embodiment can perform finer-grained alignment and fusion of two-dimensional semantic features and implicit three-dimensional geometric features, thereby obtaining an enhanced visual representation suitable for subsequent three-dimensional navigation decisions.

[0039] Specifically, a multi-head cross-modal attention mechanism is used for fine-grained alignment; let the number of attention heads be... Dimensions of each head for: For the Each attention head, calculate the query ,key Sum : in, , , For the first The learnable projection matrix corresponding to each size; No. Attention output of size for: in, This indicates a normalization operation.

[0040] In some embodiments, a pre-trained large language model is used to fuse enhanced visual representations, task language commands, and UAV state information to obtain the current time-to-time trajectory increment, including: Using large language models for language instructions Enhance visual representation and drone status Processing is performed to obtain the hidden representation. ; in, This represents the function for constructing multimodal prompt words. Represents the processing functions of a large language model; Using track regression head pairs to hide representations Processing is performed to obtain the track increment. : in, This indicates the return trajectory.

[0041] In some embodiments, determining the predicted trajectory point of the UAV at the next moment based on the current UAV trajectory point and trajectory increment includes: The predicted trajectory point of the drone at the next moment for: in, This indicates the drone's current flight path.

[0042] In some embodiments, the method further includes: The steps for jointly training the geometric prior injection model, the geometrically aware reparameterized model, the large language model, and the track regression head.

[0043] Based on the same inventive concept, embodiments of this application provide a UAV visual language navigation device based on geometric information enhancement, see reference. Figure 2 As shown, the UAV visual language navigation device 200 based on geometric information enhancement provided in this application embodiment includes at least: The acquisition unit 201 is used to acquire multi-view RGB images of the target area at the current moment, task language commands, and UAV status; The feature extraction unit 202 is used to perform bi-branch feature extraction on multi-view RGB images to obtain two-dimensional first visual semantic features and three-dimensional implicit geometric features. The first processing unit 203 is used to introduce the overall structural cues of implicit geometric features into the first visual semantic features using a pre-trained geometric prior injection model to obtain the second visual semantic features. The second processing unit 204 is used to align and fuse the second visual semantic features and implicit geometric features using a pre-trained geometric perception reparameter model to obtain an enhanced visual representation. The inference unit 205 is used to fuse the enhanced visual representation, task language commands and UAV status using a pre-trained large language model to obtain the current trajectory increment. The determining unit 206 is used to determine the predicted trajectory point of the UAV at the next moment based on the UAV trajectory point and trajectory increment at the current moment.

[0044] It should be noted that the principle of the UAV visual language navigation device 200 based on geometric information enhancement provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the UAV visual language navigation device 200 based on geometric information enhancement provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0045] Based on the same inventive concept, embodiments of this application also provide an electronic device, such as... Figure 3 As shown, it includes a memory and a processor. The memory stores an executable program, and the processor executes the executable program to implement the steps of the UAV visual language navigation method based on geometric information enhancement provided in the above embodiments.

[0046] The aforementioned processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0047] Since the electronic device described in this application embodiment is an electronic device equipped with a memory for implementing the UAV visual language navigation method based on geometric information enhancement disclosed in this application embodiment, those skilled in the art can understand the structure and variations of the electronic device described in this application embodiment based on the UAV visual language navigation method based on geometric information enhancement disclosed in this application embodiment, and therefore will not be described again here.

[0048] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the UAV visual language navigation method based on geometric information enhancement provided in the above embodiments.

[0049] The storage medium in this embodiment may be included in an electronic device; or it may exist independently and not be assembled into an electronic device. The storage medium carries one or more computer programs, which, when executed, implement the steps of the data flow behavior identification method based on multi-source logs provided in the above embodiment.

[0050] It should be understood that the various solutions in this embodiment have the same technical effects as those in the above method embodiments, and will not be repeated here.

[0051] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. Optionally, specific examples in this embodiment can refer to the examples described in any embodiment of this application, which will not be repeated here. Obviously, those skilled in the art should understand that the various modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular hardware and software combination.

[0052] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the UAV visual language navigation method based on geometric information enhancement provided in the above embodiments.

[0053] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions targeted in the blocks may occur in a different order than those targeted in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0054] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

Claims

1. A visual-language navigation method for unmanned aerial vehicles (UAVs) based on geometric information enhancement, characterized in that, include: Acquire multi-view RGB images of the target area at the current moment, task language commands, and drone status; A two-branch feature extraction method is used to extract RGB images from multiple perspectives to obtain two-dimensional first visual semantic features and three-dimensional implicit geometric features. By using a pre-trained geometric prior injection model, the overall structural cues of implicit geometric features are introduced into the first visual semantic features to obtain the second visual semantic features. The second visual semantic features and implicit geometric features are aligned and fused using a pre-trained geometric perception reparameter model to obtain an enhanced visual representation; By using a pre-trained large language model to fuse the enhanced visual representation, task language commands, and UAV status, the current trajectory increment is obtained. Based on the current drone trajectory points and track increments, the predicted drone trajectory points for the next moment are determined.

2. The method according to claim 1, characterized in that, Current moment Multi-view RGB images Represented as: in, , , , and RGB images representing the front view, rear view, left view, right view, and bottom view, respectively; Current moment drone status Represented as a six-degree-of-freedom pose : in, This indicates the position coordinates of the drone in three-dimensional space. These represent the roll angle, pitch angle, and yaw angle, respectively.

3. The method according to claim 2, characterized in that, The dual branches include a parallel Clip 2D visual encoder and a VGGT geometric encoder. A two-branch feature extraction method is used to extract RGB images from multiple perspectives to obtain two-dimensional first visual semantic features and three-dimensional implicit geometric features. include: Using the Clip 2D visual encoder to process RGB images from multiple perspectives Encoding is performed to obtain the first two-dimensional visual semantic features: ， in, Indicates Clip 2D visual encoder; Indicates the number of two-dimensional visual tokens. This represents the feature dimension output by the two-dimensional visual encoder; Using the VGGT geometric encoder to process multi-view RGB images Encoding is performed to obtain three-dimensional implicit geometric features. : ， in, This indicates the VGGT geometry encoder. Indicates the number of 3D geometry tokens. This represents the feature dimension output by the VGGT geometry encoder.

4. The method according to claim 3, characterized in that, The geometric prior injection model includes: a first branch, a second branch, and a computation unit, wherein the first branch and the second branch are parallel, and the computation unit is connected to both the first branch and the second branch. The first branch includes a first linear projection layer; the second branch includes: a second linear projection layer, an average pooling layer, two multilayer perceptrons, and a sigmoid function connected in sequence. By utilizing a pre-trained geometric prior injection model, the overall structural cues of implicit geometric features are introduced into the first visual semantic feature, resulting in the second visual semantic feature; including: The basic query representation is obtained by linearly projecting the first visual semantic features using the first linear projection layer. : in, For learnable projection matrices, For the unified feature dimensions; Using the second linear projection layer for implicit geometric features Perform linear projection to obtain features : in, For learnable projection matrices, For bias terms; Using average pooling layers for features Perform mean pooling to obtain the global geometric summary vector. ; Using two multilayer perceptrons to summarize the global geometric vector Modulation processing is performed to obtain the first scaling parameter used to modulate the two-dimensional features. and offset parameters : in, It is a two-layer, multi-layer sensor. Indicate its parameters; Using the Sigmoid function to scale the first parameter The second scaling parameter is obtained through processing. : in, Represents the Sigmoid function; Representing the basic query using computational units Second scaling parameter and offset parameters The process is performed to obtain the second visual semantic features enhanced by geometric prior. : in, This represents the learnable injection strength coefficient. This indicates element-wise multiplication.

5. The method according to claim 4, characterized in that, The geometric perception reparameter model includes: a third linear mapping layer, a fourth linear mapping layer, a multi-head attention unit, a splicing unit, a fifth linear mapping layer, and a fusion unit. The third and fourth linear mapping layers are parallel, the multi-head attention unit is connected to both the third and fourth linear mapping layers, and the multi-head attention unit, splicing unit, fifth linear mapping layer, and fusion unit are connected sequentially. An enhanced visual representation is obtained by aligning and fusing second visual semantic features and implicit geometric features using a pre-trained geometric perception reparameter model; including: Using the third linear mapping layer for implicit geometric features Perform a linear mapping to obtain key features. : in, Projection matrix, For bias terms; Using the fourth linear mapping layer for implicit geometric features Perform a linear mapping to obtain value characteristics. : in, Projection matrix, For bias terms; Using inclusion A multi-head attention unit focuses on second visual semantic features and key features. and value characteristics Processing is performed to obtain Each attention output: ; Using splicing units for all The attention outputs are concatenated to obtain the concatenated features; The fifth linear mapping layer is used to perform linear mapping on the concatenated features to obtain the aligned visual representation. ; Using fusion units to align the visual representation Compared with the basic query representation By fusing the data, an enhanced visual representation is obtained. : in, For learnable gating coefficients, .

6. The method according to claim 5, characterized in that, By fusing enhanced visual representations, task language commands, and UAV state information using a pre-trained large language model, the current time-to-time trajectory increment is obtained, including: Using large language models for language instructions Enhance visual representation and drone status Processing is performed to obtain the hidden representation. ; in, This represents the function for constructing multimodal prompt words. Represents the processing functions of a large language model; Using track regression head pairs to hide representations Processing is performed to obtain the track increment. : in, This indicates the return trajectory.

7. The method according to claim 6, characterized in that, The method further includes: The steps for jointly training the geometric prior injection model, the geometrically aware reparameterized model, the large language model, and the track regression head.

8. A visual-language navigation device for unmanned aerial vehicles (UAVs) based on geometric information enhancement, characterized in that, include: The acquisition unit is used to acquire multi-view RGB images of the target area at the current moment, task language commands, and UAV status; The feature extraction unit is used to perform bi-branch feature extraction on RGB images from multiple perspectives to obtain two-dimensional first visual semantic features and three-dimensional implicit geometric features. The first processing unit is used to introduce the overall structural cues of implicit geometric features into the first visual semantic features using a pre-trained geometric prior injection model, thereby obtaining the second visual semantic features. The second processing unit is used to align and fuse the second visual semantic features and implicit geometric features using a pre-trained geometric perception reparameter model to obtain an enhanced visual representation. The inference unit is used to fuse the enhanced visual representation, task language commands, and UAV status using a pre-trained large language model to obtain the current trajectory increment. The determination unit is used to determine the predicted trajectory point of the UAV at the next moment based on the UAV trajectory point and trajectory increment at the current moment.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-7.

Images