Multi-mode unmanned aerial vehicle intelligent navigation method and system

Abstract

The invention relates to the field of intelligent navigation of unmanned aerial vehicles, and provides a multi-mode intelligent navigation method and system for an unmanned aerial vehicle in order to solve the limitation of an existing visual language navigation method in the aspects of environment mapping and decision level splitting. The multi-mode unmanned aerial vehicle intelligent navigation method comprises the following steps: performing semantic analysis on a natural language navigation instruction to generate a multi-source semantic enhanced two-dimensional global semantic map; generating waypoint features for guiding a navigation direction; the features of the three-dimensional local semantic point cloud are extracted, cross-modal attention interaction is carried out on the features of the three-dimensional local semantic point cloud and the waypoint features to generate refined waypoint features, and then the refined waypoint features are decoded to obtain specific actions needing to be executed by the unmanned aerial vehicle at the next moment; therefore, autonomous path planning and navigation of the unmanned aerial vehicle in a complex unknown environment based on the natural language navigation instruction are realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent navigation for unmanned aerial vehicles (UAVs), and more particularly to a multimodal intelligent navigation method and system for UAVs. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The goal of UAV visual-language navigation is to enable UAVs to autonomously navigate to a target location in unknown environments based on natural language navigation commands and their own visual perception. Currently, mainstream methods mainly rely on discrete first-person visual images as observation input to guide the UAV's navigation behavior. This requires the model to have the ability to spatiotemporally correlate and semantically integrate the current field of view with past observations, and to implicitly register and fuse sequential and fragmented visual observations. However, relying solely on discrete local observations generally leads to biases in the model's understanding of the environment; moreover, different map paradigms have their own advantages and limitations. How to design a map representation method that can accurately represent the local environment, support global path planning in large-scale scenarios, and also consider memory consumption has become a current research focus.

[0004] Existing methods attempt to use large language models for high-level planning, setting sub-goals or waypoints, and then executing local navigation through a path planner strategy. However, this separate decision-making process has significant drawbacks: errors in high-level planning are directly passed to the lower levels, and environmental changes detected during lower-level execution are difficult to be fed back to the higher levels in real time for dynamic adjustments. This results in the model being unable to flexibly cope with complex and ever-changing environments, thereby reducing navigation accuracy. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a multimodal unmanned aerial vehicle (UAV) intelligent navigation method and system, which enables UAVs to perform autonomous path planning and precise navigation based on natural language navigation commands in complex and unknown environments.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a multimodal unmanned aerial vehicle (UAV) intelligent navigation method.

[0007] In one or more embodiments, a multimodal unmanned aerial vehicle (UAV) intelligent navigation method is provided, comprising: Semantic parsing is performed on natural language navigation commands to extract all semantic entity categories related to the current task. Based on this, relevant semantic targets in the UAV visual perception images are identified. Then, combined with camera parameters and depth images, a three-dimensional local semantic point cloud is constructed and projected onto a two-dimensional plane to generate a two-dimensional global semantic map with multi-source semantic enhancement. Visual feature representations of UAV visual perception images, visual feature representations of multi-source semantically enhanced 2D global semantic maps, feature representations of depth images, and semantic feature representations of natural language navigation commands are extracted and fused to generate waypoint features for guiding navigation directions. Features of 3D local semantic point clouds are extracted and interacted with waypoint features through cross-modal attention to generate refined waypoint features. The refined waypoint features are then decoded to obtain the specific actions that the UAV needs to perform in the next moment, so as to realize the autonomous path planning and navigation of UAVs based on natural language navigation commands in complex and unknown environments.

[0008] As one implementation method, the process of constructing a three-dimensional local semantic point cloud is as follows: By combining camera parameters and depth images, the pixels within any two-dimensional semantic mask at the current moment are projected into three-dimensional space to obtain the three-dimensional coordinates of all valid pixels. A unique RGB color value is assigned to each semantic category using a pre-defined color mapping function; By aggregating the 3D coordinates of all valid pixels and their corresponding semantic colors, a 3D local semantic point cloud at the current moment is obtained.

[0009] As one implementation method, an incremental fusion strategy is used to generate a two-dimensional global semantic map with multi-source semantic enhancement. The process is as follows: at each time step, the newly acquired three-dimensional local semantic point cloud is transformed to the world coordinate system and mapped to the two-dimensional plane through orthogonal projection and fused into the two-dimensional global semantic map.

[0010] As one implementation method, during the process of projecting a three-dimensional local semantic point cloud onto a two-dimensional plane, for each two-dimensional projection coordinate position, the semantic category information with the maximum height is retained as the final semantic label for that position.

[0011] As one implementation method, the current position and current fan-shaped field of view of the drone are updated and visualized in real time on a two-dimensional global semantic map, while the path points passed by the drone are accumulated and recorded to form a historical trajectory.

[0012] As one implementation method, a height information representation mechanism is introduced into the two-dimensional global semantic map. The specific process is as follows: Divide the entire two-dimensional global semantic map into a uniform grid; For each grid, iterate through all 3D local semantic point clouds within its coverage area; Record the highest height value of the point cloud in each grid area. This highest height value is marked as a label at the center of the grid and visualized in the form of a heatmap.

[0013] As one implementation method, the generated refined waypoint feature representation is as follows: : ; ; ; ; ; in, Indicates the layer normalization function; Indicates the characteristics of waypoints. Represents the attention mechanism function; , and Represents the learnable weight matrix; Represents the query vector; Represents the key vector; Represents a value vector; Indicates the dimension of the key vector; Represents the transpose of a vector; Features representing three-dimensional local semantic point clouds.

[0014] In one or more embodiments, a multimodal unmanned aerial vehicle (UAV) intelligent navigation system includes: The 2D global semantic map generation module is used to perform semantic parsing on natural language navigation instructions, extract all semantic entity categories related to the current task, identify relevant semantic targets in the UAV visual perception image, and then combine camera parameters and depth images to construct a 3D local semantic point cloud, and project it onto a 2D plane to generate a multi-source semantically enhanced 2D global semantic map. The global path planning module is used to extract visual feature representations from UAV visual perception images, visual feature representations from multi-source semantically enhanced two-dimensional global semantic maps, feature representations from depth images, and semantic feature representations from natural language navigation instructions. It then fuses these feature representations to generate waypoint features for guiding navigation directions. The local trajectory refinement module is used to extract features of the 3D local semantic point cloud, perform cross-modal attention interaction with the waypoint features to generate refined waypoint features, and then decode the refined waypoint features to obtain the specific actions that the UAV needs to perform in the next moment, so as to realize the autonomous path planning and navigation of the UAV based on natural language navigation commands in complex and unknown environments.

[0015] A third aspect of the present invention provides a computer-readable storage medium.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal unmanned aerial vehicle intelligent navigation method described above.

[0017] A fourth aspect of the present invention provides an electronic device.

[0018] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal unmanned aerial vehicle intelligent navigation method described above.

[0019] Compared with the prior art, the beneficial effects of the present invention are: To address the limitations of existing visual language navigation methods in terms of environmental mapping and the fragmentation of decision-making levels, this invention proposes a multimodal intelligent navigation method and system for unmanned aerial vehicles (UAVs). This system comprises a hybrid semantic map representation frame composed of a two-dimensional global semantic map enhanced by multi-source semantics and a three-dimensional local semantic point cloud. While balancing global environmental perception with local spatial detail representation, it effectively enhances the map's expressive power in the vertical dimension. Based on the two-dimensional global semantic map, a general navigation path is generated, and then the trajectory is refined and dynamically adjusted using the three-dimensional local semantic point cloud. This breaks through the bottleneck of unidirectional information transmission in traditional hierarchical structures, achieving bidirectional information interaction and mutual correction between high and low levels. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 This is a flowchart illustrating the multimodal unmanned aerial vehicle (UAV) intelligent navigation method according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a multimodal unmanned aerial vehicle (UAV) intelligent navigation system according to an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0025] Existing research mainly adopts the following four types of environmental map representation methods: (1) Spatial grid map: The environment is divided into grid units of fixed size. Each unit corresponds to the characteristics of a certain area in the physical space, which can capture high-density environmental information. However, this method requires preset map size, which is difficult to flexibly respond to dynamic environmental changes and has a large memory overhead; (2) Topological map: The environment layout is modeled with node-edge structure, which has the advantages of easy maintenance and low memory consumption. However, its construction usually depends on prior map information or pre-exploration stage, and is not suitable for completely unknown environments; (3) Dense 3D point cloud: It can provide rich spatial structure information. However, high memory consumption limits its application on resource-constrained platforms. (4) Semantic image mapping: By representing the semantic category of each area of ​​the environment with specific pixel colors in RGB images, intuitive visualization and compressed expression of scene information can be achieved. This type of representation works well in ground scenes, but has obvious limitations in 3D spaces with significant vertical degrees of freedom, such as UAVs: 2D images are difficult to accurately express the semantic differences at different altitude levels or flight altitudes, and cannot effectively reflect obstacles and risks in the vertical direction.

[0026] Hierarchical decision-making involves breaking down complex navigation tasks into high-level path planning and low-level action execution.

[0027] The goal of drone visual language navigation is to learn a strategy. This enables the drone intelligent agent to navigate according to natural language instructions. Perform a series of actions in an unknown three-dimensional environment. Finally, it reaches the target location described in the instruction.

[0028] Specifically, at each time step The agent receives current visual observations, including RGB images from a first-person perspective. and depth images And based on these observations and natural language navigation instructions To choose the next action, that is .

[0029] Figure 1 This is a flowchart illustrating a multimodal unmanned aerial vehicle (UAV) intelligent navigation method according to an embodiment of the present invention, as shown below. Figure 1The multimodal unmanned aerial vehicle (UAV) intelligent navigation method of this embodiment may include the following steps S101 to S103.

[0030] The specific implementation process of steps S101 to S103 is as follows: Step S101: Perform semantic parsing on the natural language navigation instructions, extract all semantic entity categories related to the current task, identify relevant semantic targets in the UAV visual perception image, and then combine camera parameters and depth images to construct a three-dimensional local semantic point cloud, and project it onto a two-dimensional plane to generate a two-dimensional global semantic map with multi-source semantic enhancement.

[0031] First, a large language model (such as GPT-4o) is invoked to process the natural language navigation instructions. Semantic parsing is performed. This process extracts a set of semantic entity categories from the instructions that are relevant to the current task and can be perceived in the physical world. : ; in, Represents a specific semantic category, ; Represents a large language model.

[0032] Based on the semantic entity categories extracted above, in this embodiment, an open vocabulary target detection model (e.g., Grounding DINO) is used to detect the front-facing camera of the drone at any given time. RGB images captured The process involves detecting bounding boxes in the image that match the target category and calculating the detection confidence and similarity to the target object for each bounding box. For targets with confidence and similarity exceeding a preset threshold, the data is further input into a lightweight instance segmentation model (e.g., MobileSAM) to perform pixel-level instance segmentation, resulting in a refined binary mask for each target. ; in, Indicates time The generated 2D semantic mask set, This represents the set of semantic category names corresponding to each mask. This indicates the Grounding DINO detection function. This represents the Mobile SAM segmentation function.

[0033] It should be noted that, in other embodiments, existing target detection models such as the YOLO series models can also be used to visually perceive images (RGB images) of the UAV. The system processes the data to detect bounding boxes in the image that match the target category.

[0034] For instance segmentation, existing image segmentation algorithms can also be used to achieve the final result of a fine binary mask for each target. Those skilled in the art can choose the appropriate algorithm based on the specific circumstances, which will not be elaborated here.

[0035] In some specific embodiments, the process of constructing a 3D local semantic point cloud is as follows: Step a1: Combine camera parameters and depth image to project any pixel in the two-dimensional semantic mask at the current moment into three-dimensional space to obtain the three-dimensional coordinates of all valid pixels; Combining camera parameters and depth images, the time... Any 2D semantic mask The pixels within the camera are projected into three-dimensional space. The three-dimensional coordinates of this pixel in the camera coordinate system are... The result can be calculated using the following inverse projection transformation formula: ; in, This indicates the coordinates of the pixel in the image coordinate system. The camera is at and Focal length in direction, These are the coordinates of the camera's principal point. This indicates that the pixel is in the depth image. The corresponding depth value.

[0036] Step a2: Use a preset color mapping function to assign a unique RGB color value to each semantic category.

[0037] Wherein, the color mapping function is ,in It is the set of all possible categories. This function is for each semantic category. Assign a unique RGB color value : .

[0038] Step a3: Aggregate the 3D coordinates of all valid pixels and their corresponding semantic colors to obtain the 3D local semantic point cloud at the current moment. .

[0039] In this embodiment, an incremental fusion strategy is used to generate a two-dimensional global semantic map with multi-source semantic enhancement. The process is as follows: at each time step, the newly acquired three-dimensional local semantic point cloud is transformed to the world coordinate system and mapped to the two-dimensional plane through orthogonal projection and fused into the two-dimensional global semantic map.

[0040] Specifically, the first step is to transform the 3D local semantic point cloud from the camera coordinate system to the world coordinate system. This process aims to register local point cloud data acquired at different times and locations to the same reference system, forming a globally consistent point cloud. The specific transformation method is as follows: ; in, This represents the coordinates of a three-dimensional point in the transformed world coordinate system. This represents the rotation matrix from the camera to the world coordinate system. It is a translation vector, representing the position of the camera in the world coordinate system.

[0041] In the process of projecting a 3D local semantic point cloud onto a 2D plane, for each 2D projection coordinate position... The semantic category information with the highest height is retained as the final semantic label for that location. : ; in represent The highest position value, This indicates the semantic category label for the corresponding position.

[0042] To further structure and denoise the semantic regions on the map, the density-based spatial clustering algorithm DBSCAN is used to aggregate regions with the same semantic category, and category text labels are added to the centroids of the top K largest clusters. Different categories of clustered regions are visualized using color coding corresponding to their categories to enhance the semantic recognizability and readability of the map.

[0043] In some alternative embodiments, the current position and current fan-shaped field of view of the UAV are updated and visualized in real time on a two-dimensional global semantic map, while the path points passed by the UAV are accumulated and recorded to form a historical trajectory.

[0044] To compensate for the shortcomings of two-dimensional maps in representing the vertical dimension, a height information representation mechanism is introduced into the two-dimensional global semantic map. The specific process is as follows: Divide the entire two-dimensional global semantic map into a uniform grid; For each grid Iterate through all three-dimensional local semantic point clouds within its coverage area; Record the highest height value of the point cloud within each grid area. : ; in A point in a 3D local semantic point cloud can be represented by the following 3D coordinates: ; Used to filter points within the grid range; This indicates the operation of taking the maximum value from the set.

[0045] The maximum altitude value is labeled at the center of the grid and visualized using a heatmap. For example, the higher the altitude value, the redder the label color; the lower the altitude value, the bluer the label color. This design provides drones with intuitive altitude-based obstacle avoidance warnings during path planning.

[0046] The altitude labels introduced in this embodiment significantly enhance the vertical expressiveness of 2D maps, making up for the inability of traditional 2D maps to accurately reflect altitude information in a 3D flight environment.

[0047] This embodiment performs semantic parsing on natural language navigation commands, extracts key target categories, guides the UAV to identify relevant semantic targets in images through the visual perception module, and constructs a 3D local semantic point cloud with semantic annotations by combining depth information. Subsequently, the point cloud is projected onto the ground plane to generate a multi-source semantically enhanced 2D global semantic map containing information such as semantic categories, historical trajectories, pose and viewpoint, and altitude heatmaps, thus constructing a hybrid semantic map system that combines global perception, local detail, and vertical expression capabilities.

[0048] Step S102: Extract the visual feature representation of the UAV visual perception image, the visual feature representation of the multi-source semantically enhanced two-dimensional global semantic map, the feature representation of the depth image, and the semantic feature representation of the natural language navigation command. Fuse the above feature representations to generate waypoint features for guiding the navigation direction.

[0049] Time intervals are extracted using a pre-trained visual encoder. RGB image and two-dimensional global semantic map Feature representation: ; in Representation of visual features of an RGB image. Visual feature representation of a two-dimensional global semantic map. This represents a pre-trained visual encoder, such as a model like ResetNet.

[0050] Use a pre-trained deep image encoder (such as RestNet) for time steps. Depth image below Processing is performed to extract depth image feature representations: ; in Representing depth image features, This represents a pre-trained deep image encoder.

[0051] Extracting natural language navigation instructions using a pre-trained text encoder Feature representation: ; in The semantic features of the instruction are represented. This refers to a pre-trained text encoder, such as BERT.

[0052] Subsequently, the four types of feature vectors are concatenated along the channel dimension to construct a unified multimodal fusion input vector: ; in This indicates a feature concatenation operation. This represents the spliced ​​multimodal features.

[0053] The above multimodal features Input to multimodal fusion module The fusion module jointly models multimodal information, extracts key patterns relevant to the navigation scenario, and outputs waypoint feature representations for directional guidance. This provides high-level semantic guidance for subsequent waypoint prediction: ; In one alternative embodiment, the multimodal fusion module A GRU architecture is adopted. To fully utilize the sequence modeling capabilities of GRU, sequences from different time points are... The sequence formed is used as input:

[0054] GRU updates the input sequence through a gating mechanism:

[0055]

[0056]

[0057] in, This represents the Sigmoid function; Represents the hyperbolic tangent activation function; It represents the Hadamardi (or Hadama) stack; , , , and Represents the learnable weight matrix; This resets the door, resetting its previous hidden state. How much of the information in it needs to be forgotten? This indicates an update gate, determining the new candidate state. Compared to the previous hidden state The fusion ratio; The candidate hidden state is determined by combining the current input with the historical state after being filtered by the reset gate. The hidden state at the current moment is a weighted combination of the previous hidden state and the candidate state. Representing the hidden state at the end of the sequence, it integrates visual, depth, and linguistic information from different time points and is used as the fused multimodal feature to provide temporal context and high-level semantic guidance for navigation decisions: .

[0058] This embodiment performs global path planning based on a two-dimensional global semantic map with multi-source semantic enhancement. It integrates two-dimensional global semantic map information, observation information, and text command information to generate waypoint features to guide navigation direction, providing high-level semantic guidance for macro-path planning and ensuring the optimality of the global path and navigation effect.

[0059] Step S103: Extract the features of the 3D local semantic point cloud, perform cross-modal attention interaction with the waypoint features to generate refined waypoint features, and then decode the refined waypoint features to obtain the specific actions that the UAV needs to perform in the next moment, so as to realize the autonomous path planning and navigation of the UAV based on natural language navigation commands in complex and unknown environments.

[0060] To accurately avoid local obstacles and adapt to complex environments, embodiments of this invention combine waypoint features with three-dimensional local semantic point clouds. Refine the process to generate a more sophisticated and executable navigation strategy.

[0061] First, the local environment is modeled, and then the PointBERT point cloud encoder is used to model it. 3D local semantic point cloud at time Encoding is performed to obtain the features of the 3D local semantic point cloud, i.e., the local environment feature vector. This feature vector can express the geometric structure and semantic information of the current local environment: ; in, This represents the pre-trained point cloud encoder PointBERT.

[0062] Map waypoint features and point cloud features to the three spaces in the attention mechanism, respectively: ; ; ; The attention calculation using the standard scaled dot product is as follows: ; The output of the attention mechanism is fused with waypoint features, and residual connections and layer normalization strategies are used to generate refined waypoint features. : ; in, Indicates the layer normalization function; Indicate waypoint characteristics; Represents the attention mechanism function; , and Represents the learnable weight matrix; Represents the query vector; Represents the key vector; Represents a value vector; Indicates the dimension of the key vector; Represents the transpose of a vector; Features representing three-dimensional local semantic point clouds.

[0063] The final refined waypoint features Input to decoder head In this context, for example, a multilayer perceptron (MLP) decodes the specific actions that the drone needs to perform in the next moment: .

[0064] This embodiment combines the detailed environmental information provided by 3D local semantic point cloud to refine waypoint features. Cross-modal attention interaction is then performed between waypoint features and 3D local semantic point cloud features to generate a refined and executable navigation strategy, improving the path accuracy and obstacle avoidance capabilities of the UAV in complex environments.

[0065] This invention designs a hierarchical navigation architecture where high-level planning and low-level execution work collaboratively. First, a general navigation path is generated based on a two-dimensional global semantic map. Then, the trajectory is refined and dynamically adjusted using a three-dimensional local semantic point cloud. This breaks through the bottleneck of unidirectional information transmission in traditional hierarchical structures, achieving bidirectional information interaction and mutual correction between high and low levels.

[0066] The multimodal UAV intelligent navigation method proposed in this invention has a higher average success rate, a higher average ideal success rate, and a lower average navigation error compared to previous technologies.

[0067] The multimodal UAV intelligent navigation method proposed in this invention has broad applications in the following areas: (1) Intelligent Unmanned Aerial Vehicle System: This invention can be applied to consumer-grade and industrial-grade intelligent unmanned aerial vehicles, enabling the unmanned aerial vehicle to understand human natural language navigation instructions and navigate autonomously in unknown environments. It is widely applicable to scenarios such as express delivery, agricultural inspection, power line inspection, and disaster search, improving the flexibility and autonomy of task execution.

[0068] (2) Intelligent robots and automated equipment: This invention is also applicable to a variety of robot systems with autonomous navigation and perception requirements, such as security patrol robots, service robots and warehouse logistics robots. By integrating visual language understanding and hybrid semantic map construction technology, the system can perform diverse tasks in complex environments, effectively improving its intelligent decision-making level and task completion efficiency.

[0069] (3) Smart City and Public Safety: This invention can be integrated into city-level unmanned inspection platforms and emergency response systems to assist in completing tasks such as urban infrastructure inspection, traffic management, accident response and regional search and rescue. It is particularly suitable for complex urban environments that require rapid deployment and dynamic response, thereby improving the level of intelligence and safety assurance capabilities of urban operations.

[0070] like Figure 2 As shown, the multimodal UAV intelligent navigation system provided in this embodiment of the invention can be implemented in software. The multimodal UAV intelligent navigation system includes the following software modules: a two-dimensional global semantic map generation module 201, a global path planning module 202, and a local trajectory refinement module 203.

[0071] The functions of each software module in the multimodal UAV intelligent navigation system are described below: The two-dimensional global semantic map generation module 201 is used to perform semantic parsing on natural language navigation instructions, extract all semantic entity categories related to the current task, identify relevant semantic targets in the UAV visual perception image, and then combine camera parameters and depth images to construct a three-dimensional local semantic point cloud, and project it onto a two-dimensional plane to generate a multi-source semantically enhanced two-dimensional global semantic map. The global path planning module 202 is used to extract visual feature representations of UAV visual perception images, visual feature representations of multi-source semantically enhanced two-dimensional global semantic maps, feature representations of depth images, and semantic feature representations of natural language navigation instructions. It then fuses these feature representations to generate waypoint features for guiding navigation directions. The local trajectory refinement module 203 is used to extract features of the three-dimensional local semantic point cloud, perform cross-modal attention interaction with the waypoint features to generate refined waypoint features, and then decode the refined waypoint features to obtain the specific actions that the UAV needs to perform in the next moment, so as to realize the autonomous path planning and navigation of the UAV based on natural language navigation commands in complex and unknown environments.

[0072] It should be noted that each module in the multimodal UAV intelligent navigation system of this invention corresponds one-to-one with each step in the multimodal UAV intelligent navigation method in the above embodiments, and their specific implementation processes are the same, so they will not be repeated here.

[0073] This invention constructs a hybrid semantic map representation consisting of a two-dimensional global semantic map with multi-source semantic enhancement and a three-dimensional local semantic point cloud. Combined with a collaborative hierarchical navigation decision-making mechanism of high-level planning and low-level execution, it achieves more accurate path planning and safe obstacle avoidance.

[0074] The structure of the electronic device according to an embodiment of the present invention will be described in detail below. Figure 3 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of the present invention. It can be understood that... Figure 3 The diagram shows only an exemplary structure of the electronic device, not the entire structure. Some or all of the structures shown may be implemented as needed.

[0075] The electronic device provided in this embodiment of the invention includes: at least one processor 301, a memory 302, a user interface 303, and at least one network interface 304. Various components in the multimodal unmanned aerial vehicle (UAV) intelligent navigation system are coupled together via a bus system 305. It can be understood that the bus system 305 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 305 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 3 The general designated all buses as Bus System 305.

[0076] The user interface 303 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0077] It is understood that memory 302 can be volatile memory or non-volatile memory, or both. In this embodiment of the invention, memory 302 is capable of storing data to support the operation of the terminal. Examples of this data include any computer programs used to operate on the terminal, such as operating systems and applications. The operating system includes various system programs, such as framework layers, core library layers, driver layers, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0078] In some embodiments, the multimodal UAV intelligent navigation system provided by the present invention can be implemented using a combination of hardware and software. For example, the multimodal UAV intelligent navigation system provided by the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the multimodal UAV intelligent navigation method provided by the present invention. For instance, the processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0079] As an example, processor 301 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0080] As an example of the hardware implementation of the multimodal UAV intelligent navigation system provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor 301 in the form of a hardware decoding processor. For example, it can be executed by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the multimodal UAV intelligent navigation method provided in this embodiment of the invention.

[0081] The memory 302 in this embodiment of the invention is used to store various types of data to support the operation of the multimodal unmanned aerial vehicle intelligent navigation system, or to store data for execution. Figure 1The program code for the method shown. Examples of this data include: any executable instructions for operation on a multimodal unmanned aerial vehicle (UAV) intelligent navigation system, such as executable instructions that can be included in the executable instructions, implementing the multimodal UAV intelligent navigation method of the embodiments of the present invention.

[0082] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including functions for executing... Figure 1 The program code for the method shown. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by the central processing unit, it performs the various functions defined in the apparatus of this application.

[0083] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal unmanned aerial vehicle (UAV) intelligent navigation method, characterized in that, include: Semantic parsing is performed on natural language navigation commands to extract all semantic entity categories related to the current task. Based on this, relevant semantic targets in the UAV visual perception images are identified. Then, combined with camera parameters and depth images, a three-dimensional local semantic point cloud is constructed and projected onto a two-dimensional plane to generate a two-dimensional global semantic map with multi-source semantic enhancement. Visual feature representations of UAV visual perception images, visual feature representations of multi-source semantically enhanced 2D global semantic maps, feature representations of depth images, and semantic feature representations of natural language navigation commands are extracted and fused to generate waypoint features for guiding navigation directions. Features of 3D local semantic point clouds are extracted and interacted with waypoint features through cross-modal attention to generate refined waypoint features. The refined waypoint features are then decoded to obtain the specific actions that the UAV needs to perform in the next moment, so as to realize the autonomous path planning and navigation of UAVs based on natural language navigation commands in complex and unknown environments.

2. The multimodal unmanned aerial vehicle (UAV) intelligent navigation method as described in claim 1, characterized in that, The process of constructing a 3D local semantic point cloud is as follows: By combining camera parameters and depth images, the pixels within any two-dimensional semantic mask at the current moment are projected into three-dimensional space to obtain the three-dimensional coordinates of all valid pixels. A unique RGB color value is assigned to each semantic category using a pre-defined color mapping function; By aggregating the 3D coordinates of all valid pixels and their corresponding semantic colors, a 3D local semantic point cloud at the current moment is obtained.

3. The multimodal unmanned aerial vehicle (UAV) intelligent navigation method as described in claim 1, characterized in that, An incremental fusion strategy is adopted to generate a two-dimensional global semantic map with multi-source semantic enhancement. The process is as follows: at each time step, the newly acquired three-dimensional local semantic point cloud is transformed to the world coordinate system and mapped to the two-dimensional plane through orthogonal projection and fused into the two-dimensional global semantic map.

4. The multimodal unmanned aerial vehicle (UAV) intelligent navigation method as described in claim 1, characterized in that, In the process of projecting a 3D local semantic point cloud onto a 2D plane, for each 2D projection coordinate position, the semantic category information with the highest height is retained as the final semantic label for that position.

5. The multimodal unmanned aerial vehicle (UAV) intelligent navigation method as described in claim 1, characterized in that, On a two-dimensional global semantic map, the current position and current fan-shaped field of view of the drone are updated and visualized in real time, while the path points of the drone are accumulated and recorded to form a historical trajectory.

6. The multimodal unmanned aerial vehicle (UAV) intelligent navigation method as described in claim 1, characterized in that, In a two-dimensional global semantic map, a height information representation mechanism is introduced, the specific process of which is as follows: Divide the entire two-dimensional global semantic map into a uniform grid; For each grid, iterate through all 3D local semantic point clouds within its coverage area; Record the highest height value of the point cloud in each grid area. This highest height value is marked as a label at the center of the grid and visualized in the form of a heatmap.

7. The multimodal unmanned aerial vehicle (UAV) intelligent navigation method as described in claim 1, characterized in that, Generate refined waypoint feature representation as : ； ； ； ； ； in, Indicates the layer normalization function; Indicates the characteristics of waypoints. Represents the attention mechanism function; Represents the query vector; Represents the key vector; Represents a value vector; Indicates the dimension of the key vector; Represents the transpose of a vector; Features representing three-dimensional local semantic point clouds.

8. A multimodal unmanned aerial vehicle (UAV) intelligent navigation system, characterized in that, include: The 2D global semantic map generation module is used to perform semantic parsing on natural language navigation instructions, extract all semantic entity categories related to the current task, identify relevant semantic targets in the UAV visual perception image, and then combine camera parameters and depth images to construct a 3D local semantic point cloud, and project it onto a 2D plane to generate a multi-source semantically enhanced 2D global semantic map. The global path planning module is used to extract visual feature representations from UAV visual perception images, visual feature representations from multi-source semantically enhanced two-dimensional global semantic maps, feature representations from depth images, and semantic feature representations from natural language navigation instructions. It then fuses these feature representations to generate waypoint features for guiding navigation directions. The local trajectory refinement module is used to extract features of the 3D local semantic point cloud, perform cross-modal attention interaction with the waypoint features to generate refined waypoint features, and then decode the refined waypoint features to obtain the specific actions that the UAV needs to perform in the next moment, so as to realize the autonomous path planning and navigation of the UAV based on natural language navigation commands in complex and unknown environments.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multimodal unmanned aerial vehicle intelligent navigation method as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multimodal unmanned aerial vehicle intelligent navigation method as described in any one of claims 1-7.

Images