A multi-modal unmanned aerial vehicle intelligent navigation method and system

Abstract

The present application relates to the field of unmanned aerial vehicle intelligent navigation, and provides a multi-modal unmanned aerial vehicle intelligent navigation method and system to solve the limitations of existing visual language navigation methods in environment mapping and decision level fragmentation. The multi-modal unmanned aerial vehicle intelligent navigation method includes semantic analysis of natural language navigation instructions, generation of a multi-source semantic enhanced two-dimensional global semantic map, generation of a waypoint feature for guiding navigation direction, extraction of a three-dimensional local semantic point cloud feature, cross-modal attention interaction of the three-dimensional local semantic point cloud feature with the waypoint feature, generation of a refined waypoint feature, decoding of the refined waypoint feature, and obtaining of specific actions to be performed by the unmanned aerial vehicle at the next time, to realize autonomous path planning and navigation of the unmanned aerial vehicle in a complex unknown environment based on natural language navigation instructions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned aerial vehicle intelligent navigation, and in particular to a multi-modal unmanned aerial vehicle intelligent navigation method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] The goal of visual language navigation for unmanned aerial vehicles is to enable unmanned aerial vehicles to navigate autonomously to a target location in an unknown environment according to natural language navigation instructions combined with their own visual perception. Currently, mainstream methods mainly rely on discrete first-person visual images as observation input to guide the navigation behavior of unmanned aerial vehicles. This requires the model to have the ability to spatiotemporally correlate the current field of view with past observations and integrate semantics, and to implicitly register and fuse sequential and fragmented visual observations. However, relying solely on discrete local observations generally leads to biased understanding of the environment by the model; and different map paradigms have their own advantages and limitations. How to design a map representation method that can accurately represent the local environment, support global path planning in large-scale scenarios, and also take into account memory consumption has become a research focus.

[0004] Existing methods attempt to use large language models for high-level planning, set sub-goals or waypoints, and then use a path planner to execute local navigation. However, this separate decision-making process has obvious defects. Errors in high-level planning are directly passed to the lower level, and environmental changes found in lower-level execution are difficult to feedback to the high level in real time for dynamic adjustment, resulting in the model being unable to flexibly respond to complex and changing environments, thereby reducing navigation accuracy. SUMMARY

[0005] To solve the above technical problems, the present application provides a multi-modal unmanned aerial vehicle intelligent navigation method and system, which can realize autonomous path planning and accurate navigation of unmanned aerial vehicles in complex unknown environments based on natural language navigation instructions.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] The first aspect of the present application provides a multi-modal unmanned aerial vehicle intelligent navigation method.

[0008] In one or more embodiments, a multi-modal unmanned aerial vehicle intelligent navigation method is provided, comprising:

[0009] The natural language navigation instruction is semantically analyzed, all semantic entity categories related to the current task are extracted, relevant semantic targets in the visual perception image of the unmanned aerial vehicle are identified according to the semantic entity categories, a three-dimensional local semantic point cloud is constructed by combining the camera parameters and the depth image, and the three-dimensional local semantic point cloud is projected onto a two-dimensional plane to generate a two-dimensional global semantic map enhanced by multiple sources of semantics.

[0010] The visual feature representation of the visual perception image of the unmanned aerial vehicle, the visual feature representation of the two-dimensional global semantic map enhanced by multiple sources of semantics, the feature representation of the depth image, and the semantic feature representation of the natural language navigation instruction are extracted, the feature representations are fused, and waypoint features for guiding navigation directions are generated.

[0011] The features of the three-dimensional local semantic point cloud are extracted, the features are subjected to cross-modal attention interaction with the waypoint features, refined waypoint features are generated, and the refined waypoint features are decoded to obtain specific actions to be performed by the unmanned aerial vehicle at the next moment, so as to realize autonomous path planning and navigation of the unmanned aerial vehicle in a complex unknown environment based on the natural language navigation instruction.

[0012] As an implementation mode, the process of constructing the three-dimensional local semantic point cloud is as follows:

[0013] The pixel points in any two-dimensional semantic mask at the current moment are projected into a three-dimensional space by combining the camera parameters and the depth image, and three-dimensional coordinates of all effective pixel points are obtained.

[0014] A preset color mapping function is used to assign a unique RGB color value to each semantic category.

[0015] The three-dimensional coordinates of all effective pixel points and the corresponding semantic colors are aggregated to obtain the three-dimensional local semantic point cloud at the current moment.

[0016] As an implementation mode, an incremental fusion strategy is used to generate the two-dimensional global semantic map enhanced by multiple sources of semantics, and the process is as follows: at each time step, the newly obtained three-dimensional local semantic point cloud is converted to a world coordinate system and mapped to a two-dimensional plane by orthogonal projection and fused into the two-dimensional global semantic map.

[0017] As an implementation mode, in the process of projecting the three-dimensional local semantic point cloud onto a two-dimensional plane, for each two-dimensional projection coordinate position, the semantic category information with the maximum height is retained as the final semantic label of the position.

[0018] As an implementation mode, on the two-dimensional global semantic map, the current position and the current fan-shaped visual angle range of the unmanned aerial vehicle are updated and visualized in real time, and the path points flown over by the unmanned aerial vehicle are accumulated and recorded to form a historical trajectory.

[0019] As an implementation, in the two-dimensional global semantic map, a height information representation mechanism is introduced, and the specific process is as follows:

[0020] The entire two-dimensional global semantic map is divided into uniform grids;

[0021] For each grid, all three-dimensional local semantic point clouds in its coverage range are traversed;

[0022] The highest height value of the point cloud in each grid area is recorded, and the highest height value is marked as a label at the center position of the grid, and a heat map form is used for visualization.

[0023] As an implementation, the fine navigation point feature representation is generated as :

[0024] ;

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] wherein, represents a layer normalization function; represents a navigation point feature. represents an attention mechanism function; , and represent a learnable weight matrix; represents a query vector; represents a key vector; represents a value vector; represents the dimension of the key vector; represents the transpose of the vector; represents the feature of the three-dimensional local semantic point cloud.

[0030] In one or more embodiments, a multi-modal unmanned aerial vehicle intelligent navigation system comprises:

[0031] A two-dimensional global semantic map generation module is used for semantic analysis of natural language navigation instructions, extracting all semantic entity categories related to the current task, identifying relevant semantic targets in the unmanned aerial vehicle visual perception image, and combining camera parameters and depth images to construct three-dimensional local semantic point clouds, and projecting them to a two-dimensional plane to generate a multi-source semantic enhanced two-dimensional global semantic map;

[0032] a global path planning module configured to extract a visual feature representation of a visual perception image of the UAV, a visual feature representation of a multi-source semantic enhanced two-dimensional global semantic map, a feature representation of a depth image, and a semantic feature representation of a natural language navigation instruction, fuse the feature representations, and generate waypoint features for guiding navigation directions;

[0033] a local trajectory refinement module configured to extract features of a three-dimensional local semantic point cloud, perform cross-modal attention interaction between the features and the waypoint features, generate refined waypoint features, and decode the refined waypoint features to obtain specific actions to be performed by the UAV at a next time instant, so as to realize autonomous path planning and navigation of the UAV in a complex unknown environment based on the natural language navigation instruction.

[0034] A third aspect of the present application provides a computer-readable storage medium.

[0035] A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps in the multi-modal UAV intelligent navigation method described above.

[0036] A fourth aspect of the present application provides an electronic device.

[0037] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the multi-modal UAV intelligent navigation method described above when executing the program.

[0038] Compared with the prior art, the present application has the following advantages:

[0039] The present application is to solve the limitations of existing visual language navigation methods in the aspects of environment mapping and decision level fragmentation, and proposes a multi-modal UAV intelligent navigation method and system, which comprises a hybrid semantic map representation framework composed of a multi-source semantic enhanced two-dimensional global semantic map and a three-dimensional local semantic point cloud, which effectively enhances the expression ability of the map in the vertical dimension while taking into account global environment perception and local spatial detail expression; based on the two-dimensional global semantic map, a rough navigation path is generated, and then the three-dimensional local semantic point cloud is used to refine and dynamically adjust the trajectory, breaking the one-way bottleneck of information transmission in the traditional hierarchical structure, and realizing bidirectional information interaction and mutual correction of high and low levels. BRIEF DESCRIPTION OF DRAWINGS

[0040] The drawings accompanying the specification of the present application form a part thereof, serve to provide further understanding of the present application, and together with the exemplary embodiments of the present application and their description, serve to explain the present application, and do not constitute an improper limitation of the present application.

[0041] Figure 1A flowchart of a multi-modal unmanned aerial vehicle intelligent navigation method according to an embodiment of the present application;

[0042] Figure 2 A structural diagram of a multi-modal unmanned aerial vehicle intelligent navigation system according to an embodiment of the present application;

[0043] Figure 3 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] The present application will be further described below with reference to the accompanying drawings and embodiments.

[0045] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application pertains.

[0046] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a reference to the presence of a feature, step, operation, device, component, and / or combinations thereof.

[0047] Existing research mainly adopts the following four types of environment map representation methods: (1) Spatial grid map: the environment is divided into fixed-size grid cells, each cell corresponds to the characteristics of a certain area in the physical space, which can capture high-density environment information. However, this method needs to pre-set the map size, which is difficult to flexibly respond to dynamic environmental changes, and has a large memory overhead; (2) Topological map: the environment layout is modeled as a node-edge structure, which has the advantages of easy maintenance and low memory occupancy, but its construction usually depends on prior map information or pre-exploration stage, which is not suitable for completely unknown environments; (3) Dense three-dimensional point cloud: it can provide rich spatial structure information, but high memory consumption limits its application on resource-constrained platforms. (4) Semantic image mapping: by representing the semantic categories of each region in the environment with a specific pixel color in the RGB image, the scene information is visualized and compressed expressed. This type of representation works well in ground scenes, but has obvious limitations in three-dimensional space with significant vertical freedom such as unmanned aerial vehicles: two-dimensional images cannot accurately represent semantic differences at different height levels or flight altitudes, and cannot effectively reflect obstacles and risks in the vertical direction.

[0048] Hierarchical decision-making, i.e., decomposing complex navigation tasks into high-level path planning and low-level action execution.

[0049] The goal of visual language navigation for drones is to learn a policy that makes the drone agent follow natural language navigation instructions in an unknown three-dimensional environment and eventually reach the target location described by the instructions.

[0050] Specifically, at each time step the agent receives the current visual observations, including a first-person perspective RGB image and a depth image and selects the next action based on these observations and the natural language navigation instructions . .

[0051] Figure 1 is a flowchart of a multi-modal intelligent navigation method for drones according to an embodiment of the present application. As shown in the multi-modal intelligent navigation method for drones according to the embodiment, the method can include the following steps S101-S103. Figure 1

[0052] The specific implementation process of steps S101-S103 is as follows:

[0053] Step S101: Perform semantic analysis on the natural language navigation instructions, extract all semantic entity categories related to the current task, identify the relevant semantic targets in the visual perception image of the drone according to the semantic entity categories, and then combine the camera parameters and the depth image to construct a three-dimensional local semantic point cloud, and project it to a two-dimensional plane to generate a multi-source semantic enhanced two-dimensional global semantic map.

[0054] First, call a large language model (such as GPT-4o) to perform semantic analysis on the natural language navigation instructions. This process extracts a set of semantic entity categories related to the current task from the instructions that can be perceived in the physical world:

[0055]

[0056] wherein represents a specific semantic category ; and represents the large language model.

[0057] Based on the above extracted semantic entity categories, in this embodiment, an open vocabulary object detection model (such as Grounding DINO) is used to detect the RGB image captured by the front camera of the drone at time ​​​​​The processing is performed, the bounding box in the image matching the target category is detected, and the detection confidence and the similarity to the target object of each bounding box are calculated. For the target whose confidence and similarity exceed the preset threshold, a lightweight instance segmentation model (such as MobileSAM) is further input to perform a pixel-level instance segmentation operation to obtain a fine binary mask of each target:

[0058] ;

[0059] wherein, represents the time generated 2D semantic mask set, represents a set of semantic category names corresponding to each mask, represents a grounding DINO detection function, represents a Mobile SAM segmentation function.

[0060] It should be noted that in other embodiments, an existing target detection model such as a YOLO series model can also be used to process the unmanned aerial vehicle visual perception image (RGB image ) to detect the bounding box in the image matching the target category.

[0061] For the instance segmentation operation, an existing image segmentation algorithm can also be used to implement, and finally a fine binary mask of each target is obtained. Those skilled in the art can select according to the actual situation, which will not be described in detail here.

[0062] In some specific embodiments, the process of constructing a three-dimensional local semantic point cloud is as follows:

[0063] Step a1: combining camera parameters and depth images, projecting the pixel points in any one two-dimensional semantic mask at the current time into a three-dimensional space to obtain the three-dimensional coordinates of all effective pixel points;

[0064] Combining camera parameters and depth images, projecting the pixel points in any one 2D semantic mask at time into a three-dimensional space. The three-dimensional coordinates of the pixel point in the camera coordinate system can be calculated by the following inverse projection transformation formula:

[0065] ;

[0066] wherein, represents the coordinates of the pixel point in the image coordinate system, are the focal lengths of the camera in the and directions, is the principal point coordinate of the camera, ​represents the corresponding depth value of the pixel point in the depth image.

[0067] Step a2: Assign a unique RGB color value to each semantic class by using a pre-defined color mapping function.

[0068] wherein the color mapping function is wherein is the set of all possible classes. The function assigns a unique RGB color value to each semantic class

[0069]

[0070] Step a3: Aggregate the three-dimensional coordinates of all valid pixel points and their corresponding semantic colors to obtain the three-dimensional local semantic point cloud at the current time

[0071] In this embodiment, an incremental fusion strategy is adopted to generate a multi-source semantic-enhanced two-dimensional global semantic map. The process is as follows: at each time step, the newly acquired three-dimensional local semantic point cloud is converted to the world coordinate system and mapped to the two-dimensional plane by orthogonal projection and fused into the two-dimensional global semantic map.

[0072] Specifically, the three-dimensional local semantic point cloud in the camera coordinate system is first converted to the world coordinate system. This process aims to register the local point cloud data collected at different times and different positions to the same reference system to form a globally consistent point cloud. The specific transformation method is as follows:

[0073]

[0074] wherein represents the three-dimensional point coordinates in the world coordinate system after conversion, represents the rotation matrix from the camera to the world coordinate system, is the translation vector, representing the position of the camera in the world coordinate system.

[0075] In the process of projecting the three-dimensional local semantic point cloud to the two-dimensional plane, for each two-dimensional projection coordinate position , the semantic class information with the maximum height is retained as the final semantic label of the position:

[0076]

[0077] wherein represents the highest value at the position, ​​​​​​​​representing a semantic category label of a corresponding position.

[0078] To further structure and denoise the semantic regions on the map, a density-based spatial clustering of applications with noise (DBSCAN) algorithm is used to aggregate regions of the same semantic category, and a category label is added at the position of the centroid of the top K largest clusters. The clustered regions of different categories are visualized using color coding corresponding to the categories to enhance the semantic recognition and readability of the map.

[0079] In some optional embodiments, on the two-dimensional global semantic map, the current position and the current fan-shaped visual angle range of the UAV are updated and visualized in real time, and the path points flown through by the UAV are accumulated and recorded to form a historical trajectory.

[0080] To compensate for the lack of vertical dimension expression on the two-dimensional map, a height information representation mechanism is introduced in the two-dimensional global semantic map, and the specific process is as follows:

[0081] The entire two-dimensional global semantic map is divided into uniform grids;

[0082] For each grid , all three-dimensional local semantic point clouds in its coverage range are traversed;

[0083] The highest height value of the point cloud in each grid region is recorded :

[0084] ;

[0085] wherein represents a point in the three-dimensional local semantic point cloud, and the three-dimensional coordinates of the point can be represented as ; is used to filter out points within the grid range; represents a maximum value operation from the set.

[0086] The highest height value is labeled as a label at the center position of the grid and visualized in the form of a heat map. For example, the higher the height value, the more the label color tends to be red; the lower the height value, the more the label color tends to be blue. This design provides an intuitive height obstacle avoidance warning for the UAV when planning a path.

[0087] The height label introduced in this embodiment significantly enhances the expression ability of the two-dimensional map in the vertical direction, and compensates for the lack of traditional two-dimensional maps in accurately reflecting height information in a three-dimensional flight environment.

[0088] This embodiment performs semantic parsing on natural language navigation commands, extracts key target categories, guides the UAV to identify relevant semantic targets in images through the visual perception module, and constructs a 3D local semantic point cloud with semantic annotations by combining depth information. Subsequently, the point cloud is projected onto the ground plane to generate a multi-source semantically enhanced 2D global semantic map containing information such as semantic categories, historical trajectories, pose and viewpoint, and altitude heatmaps, thus constructing a hybrid semantic map system that combines global perception, local detail, and vertical expression capabilities.

[0089] Step S102: Extract the visual feature representation of the UAV visual perception image, the visual feature representation of the multi-source semantically enhanced two-dimensional global semantic map, the feature representation of the depth image, and the semantic feature representation of the natural language navigation command. Fuse the above feature representations to generate waypoint features for guiding the navigation direction.

[0090] Time intervals are extracted using a pre-trained visual encoder. RGB image and two-dimensional global semantic map Feature representation:

[0091] ;

[0092] in Representation of visual features of an RGB image. Visual feature representation of a two-dimensional global semantic map. This represents a pre-trained visual encoder, such as a model like ResetNet.

[0093] Use a pre-trained deep image encoder (such as RestNet) for time steps. Depth image below Processing is performed to extract depth image feature representations:

[0094] ;

[0095] in Representing depth image features, This represents a pre-trained deep image encoder.

[0096] Extracting natural language navigation instructions using a pre-trained text encoder Feature representation:

[0097] ;

[0098] in This represents the semantic features of the instruction. This refers to a pre-trained text encoder, such as BERT.

[0099] Subsequently, the four types of feature vectors are concatenated along the channel dimension to construct a unified multimodal fusion input vector:

[0100] ;

[0101] in This indicates a feature concatenation operation. This represents the spliced ​​multimodal features.

[0102] The above multimodal features Input to multimodal fusion module The fusion module jointly models multimodal information, extracts key patterns relevant to the navigation scenario, and outputs waypoint feature representations for directional guidance. This provides high-level semantic guidance for subsequent waypoint prediction:

[0103] ;

[0104] In one alternative embodiment, the multimodal fusion module A GRU architecture is adopted. To fully utilize the sequence modeling capabilities of GRU, sequences from different time points are... The sequence formed is used as input:

[0105]

[0106] GRU updates the input sequence through a gating mechanism:

[0107]

[0108]

[0109]

[0110] in, This represents the Sigmoid function; Represents the hyperbolic tangent activation function; It represents the Hadamardi (or Hadama) stack; , , , and Represents the learnable weight matrix; This resets the door, resetting its previous hidden state. How much of the information in it needs to be forgotten? This indicates an update gate, determining the new candidate state. Compared to the previous hidden state The fusion ratio; For the candidate hidden state, the current input is combined with the historical state filtered by the reset gate; For the current hidden state, a weighted combination of the previous hidden state and the candidate state is used. The hidden state at the end of the sequence, which integrates visual, depth and language information at different time points, is used as the fused multi-modal feature, providing temporal context and high-level semantic guidance for navigation decision-making: .

[0111] The embodiment is based on a two-dimensional global semantic map enhanced by multi-source semantics for global path planning, which integrates two-dimensional global semantic map information, observation information and text instruction information to generate waypoint features for guiding navigation directions, providing high-level semantic guidance for macro path planning and ensuring the optimality of global path and navigation effect.

[0112] Step S103: Extract the features of the three-dimensional local semantic point cloud, and perform cross-modal attention interaction with the waypoint features to generate refined waypoint features, and then decode the refined waypoint features to obtain the specific actions to be performed by the unmanned aerial vehicle at the next time, so as to realize autonomous path planning and navigation of the unmanned aerial vehicle in a complex unknown environment based on natural language navigation instructions.

[0113] In order to accurately avoid local obstacles and adapt to complex environments, the embodiment of the present application refines the waypoint features combined with the three-dimensional local semantic point cloud to generate more refined and executable navigation strategies.

[0114] First, the local environment is modeled, and the point cloud encoder PointBERT is used to encode the three-dimensional local semantic point cloud at the current time to obtain the features of the three-dimensional local semantic point cloud, i.e. the local environment feature vector , which can express the geometric structure and semantic information in the current local environment:

[0115] ;

[0116] wherein, PointBERT represents a pre-trained point cloud encoder.

[0117] The waypoint feature and the point cloud feature are respectively mapped to three spaces in the attention mechanism:

[0118] ;

[0119] ;

[0120] ; ​

[0121] The standard scaled dot-product attention is calculated as follows:

[0122]

[0123] The output of the attention mechanism is fused with the waypoint features, and a residual connection and layer normalization strategy are adopted to generate refined waypoint features

[0124]

[0125] wherein, denotes a layer normalization function; denotes a waypoint feature; denotes an attention mechanism function; , and denote learnable weight matrices; denotes a query vector; denotes a key vector; denotes a value vector; denotes the dimension of the key vector; denotes the transpose of a vector; denotes the feature of a three-dimensional local semantic point cloud.

[0126] The final refined waypoint features are input into a decoding head , such as a multi-layer perceptron (MLP), to decode specific actions to be performed by the UAV at the next time step: .

[0127] The embodiment refines the waypoint features in combination with the refined environmental information provided by the three-dimensional local semantic point cloud. Cross-modal attention interaction is performed between the waypoint features and the three-dimensional local semantic point cloud features to generate a refined and executable navigation strategy, thereby improving the path accuracy and obstacle avoidance capability of the UAV in a complex environment.

[0128] The present application designs a hierarchical navigation architecture in which high-level planning and low-level execution work collaboratively. A rough navigation path is generated based on a two-dimensional global semantic map, and the trajectory is refined and dynamically adjusted with the aid of a three-dimensional local semantic point cloud. The bottleneck of one-way information transmission in the traditional hierarchical structure is broken, and bidirectional information interaction and mutual correction between high and low levels are achieved.

[0129] Compared with the prior art, the multi-modal UAV intelligent navigation method has higher average success rate, higher average ideal success rate, and lower average navigation error.

[0130] The multi-modal UAV intelligent navigation method can be widely applied in the following scenarios:​​​

[0131] (1) Intelligent unmanned aerial vehicle system: the application can be applied to consumer and industrial intelligent unmanned aerial vehicles, enabling the unmanned aerial vehicle to understand natural language navigation instructions, autonomously navigate in unknown environments, and be widely used in scenarios such as express delivery, agricultural inspection, power line inspection, disaster search, and the like, thereby improving the flexibility and autonomy of task execution.

[0132] (2) Intelligent robots and automated equipment: the application is also applicable to various robot systems with autonomous navigation and perception needs, such as security patrol robots, service robots, and warehouse logistics robots, and the like, and by fusing visual language understanding and hybrid semantic map construction technology, the system can perform diversified tasks in complex environments, thereby effectively improving the intelligent decision-making level and task completion efficiency.

[0133] (3) Smart city and public safety: the application can be integrated into city-level unmanned inspection platforms and emergency response systems to assist in completing tasks such as city infrastructure inspection, traffic management, accident response, and regional search and rescue, and is particularly suitable for complex urban environments that require rapid deployment and dynamic response, thereby improving the intelligent level and safety guarantee capability of city operation.

[0134] As shown in Figure 2 , the multi-modal unmanned aerial vehicle intelligent navigation system provided by the embodiment of the application can be implemented in a software manner, and the multi-modal unmanned aerial vehicle intelligent navigation system includes the following software modules: a two-dimensional global semantic map generation module 201, a global path planning module 202, and a local trajectory refinement module 203.

[0135] The functions of each software module in the multi-modal unmanned aerial vehicle intelligent navigation system will be introduced as follows:

[0136] The two-dimensional global semantic map generation module 201 is used for semantic analysis of natural language navigation instructions, extraction of all semantic entity categories related to the current task, identification of related semantic targets in the visual perception image of the unmanned aerial vehicle, construction of a three-dimensional local semantic point cloud in combination with camera parameters and a depth image, and projection of the three-dimensional local semantic point cloud to a two-dimensional plane to generate a two-dimensional global semantic map enhanced by multiple sources of semantics;

[0137] The global path planning module 202 is used for extraction of visual feature representations of the visual perception image of the unmanned aerial vehicle, visual feature representations of the two-dimensional global semantic map enhanced by multiple sources of semantics, feature representations of the depth image, and semantic feature representations of the natural language navigation instructions, fusion of the above feature representations, and generation of a waypoint feature for guiding a navigation direction;

[0138] The local trajectory refinement module 203 is configured to extract features of the three-dimensional local semantic point cloud, perform cross-modal attention interaction with the waypoint features, generate refined waypoint features, and decode the refined waypoint features to obtain specific actions to be performed by the UAV at the next moment, so as to realize autonomous path planning and navigation of the UAV in a complex unknown environment based on natural language navigation instructions.

[0139] It should be noted that each module in the multi-modal UAV intelligent navigation system of the embodiment of the present application corresponds to each step in the multi-modal UAV intelligent navigation method of the above embodiment one by one, and the specific implementation process is the same, which will not be repeated here.

[0140] The present application realizes more accurate path planning and safe obstacle avoidance by constructing a hybrid semantic map representation composed of a multi-source semantic enhanced two-dimensional global semantic map and a three-dimensional local semantic point cloud, and combining a collaborative hierarchical navigation decision mechanism of high-level planning and low-level execution.

[0141] The structure of the electronic device of the embodiment of the present application will be described in detail below. Figure 3 The composition structure diagram of the electronic device provided by the embodiment of the present application can be understood as follows. Figure 3 Only exemplary structures of the electronic device are shown, not all structures, and part or all of the structures shown can be implemented as needed.

[0142] The electronic device provided by the embodiment of the present application includes at least one processor 301, a memory 302, a user interface 303 and at least one network interface 304. Each component in the multi-modal UAV intelligent navigation system is coupled together through a bus system 305. It can be understood that the bus system 305 is used to realize the connection and communication between these components. In addition to including a data bus, the bus system 305 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, all kinds of buses are marked as bus system 305 in the Figure 3

[0143] The user interface 303 can include a display, a keyboard, a mouse, a trackball, a click wheel, a key, a button, a touchpad or a touch screen, etc.

[0144] It can be understood that the memory 302 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. The memory 302 in the embodiment of the present application can store data to support the operation of the terminal. Examples of these data include any computer programs for operating on the terminal, such as an operating system and an application program. The operating system contains various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application program can include various application programs.​

[0145] In some embodiments, the multi-modal unmanned aerial vehicle intelligent navigation system provided by the embodiments of the present application can be implemented in a combination of software and hardware. For example, the multi-modal unmanned aerial vehicle intelligent navigation system provided by the embodiments of the present application can be a hardware decoding processor, which is programmed to execute the multi-modal unmanned aerial vehicle intelligent navigation method provided by the embodiments of the present application. For example, the hardware decoding processor can be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0146] For example, the processor 301 can be an integrated circuit chip with a processing capability of signals, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0147] As an example of the multi-modal unmanned aerial vehicle intelligent navigation system provided by the embodiments of the present application implemented in hardware, the apparatus provided by the embodiments of the present application can directly use a hardware decoding processor in the form of a processor 301 to complete the execution, such as one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components to execute the multi-modal unmanned aerial vehicle intelligent navigation method provided by the embodiments of the present application.

[0148] The memory 302 in the embodiments of the present application is used to store various types of data to support the operation of the multi-modal unmanned aerial vehicle intelligent navigation system, or to store instructions for execution by the processor 301 to perform the multi-modal unmanned aerial vehicle intelligent navigation method provided by the embodiments of the present application. Figure 1program code of the method. Examples of such data include any executable code such as an executable instruction for operating on a multi-modal UAV intelligent navigation system, and a program implementing the multi-modal UAV intelligent navigation method of the embodiments of the present application can be included in the executable instruction.

[0149] In particular, the processes described above with reference to the flow charts can be implemented as computer software programs in accordance with the embodiments of the present application. For example, the embodiments of the present application include a computer program product comprising a computer program which contains instructions for executing the multi-modal UAV intelligent navigation method of the embodiments of the present application, and the computer program can be installed in a computer readable medium. Figure 1 program code of the method. In such embodiments, the computer program can be downloaded and installed from a network by a communication section, and / or installed from a detachable medium. When the computer program is executed by the central processing unit, various functions defined in the apparatus of the present application are executed.

[0150] The present application is described with reference to the flow charts and / or block diagrams of the methods, apparatus (system) and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flow charts and / or block diagrams, and combinations of flows and / or blocks in the flow charts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, generate a means for implementing the functions specified in the flow charts and / or block diagrams of the embodiments of the present application. Figure 1 one flow or multiple flows and / or blocks Figure 1 Figure 1 an apparatus for implementing the functions specified in one block or multiple blocks.

[0151] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-modal unmanned aerial vehicle intelligent navigation method, characterized in that, Comprise: The semantic analysis of natural language navigation instruction is carried out, all semantic entity categories related to the current task are extracted, the relevant semantic targets in the visual perception image of the unmanned aerial vehicle are identified according to the semantic entity categories, the three-dimensional local semantic point cloud is constructed by combining the camera parameters and the depth image, and the three-dimensional local semantic point cloud is projected onto the two-dimensional plane to generate the two-dimensional global semantic map enhanced by multiple sources of semantics; The visual feature representation of the visual perception image of the unmanned aerial vehicle, the visual feature representation of the two-dimensional global semantic map enhanced by multiple sources of semantics, the feature representation of the depth image and the semantic feature representation of the natural language navigation instruction are extracted, the four feature representations are fused, and the waypoint feature for guiding the navigation direction is generated; The features of the three-dimensional local semantic point cloud are extracted, the three-dimensional local semantic point cloud is interacted with the waypoint feature through cross-modal attention, the refined waypoint feature is generated, and the specific action to be performed by the unmanned aerial vehicle at the next moment is obtained by decoding the refined waypoint feature, so that the autonomous path planning and navigation of the unmanned aerial vehicle in a complex unknown environment based on the natural language navigation instruction are realized.

2. The multi-modal drone intelligent navigation method of claim 1, wherein, The process of constructing the three-dimensional local semantic point cloud is: The pixel points in any two-dimensional semantic mask at the current moment are projected into three-dimensional space by combining the camera parameters and the depth image, and the three-dimensional coordinates of all effective pixel points are obtained; A unique RGB color value is assigned to each semantic category by using a preset color mapping function; The three-dimensional coordinates of all effective pixel points and the corresponding semantic colors are aggregated to obtain the three-dimensional local semantic point cloud at the current moment.

3. The multi-modal drone intelligent navigation method of claim 1, wherein, An incremental fusion strategy is adopted to generate the two-dimensional global semantic map enhanced by multiple sources of semantics, and the process is as follows: at each time step, the newly obtained three-dimensional local semantic point cloud is converted to the world coordinate system and mapped to the two-dimensional plane through orthogonal projection and fused into the two-dimensional global semantic map.

4. The multi-modal drone intelligent navigation method of claim 1, wherein, In the process of projecting the three-dimensional local semantic point cloud onto the two-dimensional plane, for each two-dimensional projection coordinate position, the semantic category information with the maximum height is retained as the final semantic label of the position.

5. The multi-modal drone intelligent navigation method of claim 1, wherein, On the two-dimensional global semantic map, the current position and the current fan-shaped visual angle range of the unmanned aerial vehicle are updated and visualized in real time, and the path points flown by the unmanned aerial vehicle are recorded to form a historical trajectory.

6. The multi-modal drone intelligent navigation method of claim 1, wherein, In the two-dimensional global semantic map, a height information representation mechanism is introduced, and the specific process is as follows: The entire two-dimensional global semantic map is divided into uniform grids; For each grid, all three-dimensional local semantic point clouds in its coverage range are traversed; The highest height value of the point cloud in each grid region is recorded, and the highest height value is marked as a label at the center position of the grid, and a heat map is used for visualization.

7. The multi-modal drone intelligent navigation method of claim 1, wherein, generating a refined waypoint feature representation as : ； ； ； ； ； wherein, denotes a layer normalization function; denotes a waypoint feature; denotes an attention mechanism function; denotes a query vector; denotes a key vector; denotes a value vector; denotes a dimension of a key vector; denotes a transpose of a vector; denotes a feature of a three-dimensional local semantic point cloud.

8. A multi-modal drone intelligent navigation system, characterized in that, Comprise: The two-dimensional global semantic map generation module is used for semantic analysis of natural language navigation instruction, extracting all semantic entity categories related to the current task, identifying relevant semantic targets in the visual perception image of the unmanned aerial vehicle according to the semantic entity categories, constructing three-dimensional local semantic point cloud by combining camera parameters and depth image, and projecting the three-dimensional local semantic point cloud onto the two-dimensional plane to generate the two-dimensional global semantic map enhanced by multiple sources of semantics; a global path planning module configured to extract a visual feature representation of a visual perception image of the UAV, a visual feature representation of a multi-source semantic enhanced two-dimensional global semantic map, a feature representation of a depth image, and a semantic feature representation of a natural language navigation instruction, fuse the four feature representations, and generate waypoint features for guiding navigation directions; a local trajectory refinement module configured to extract features of a three-dimensional local semantic point cloud, perform cross-modal attention interaction between the features and the waypoint features, generate refined waypoint features, and decode the refined waypoint features to obtain specific actions to be performed by the UAV at a next time, so as to realize autonomous path planning and navigation of the UAV in a complex unknown environment based on the natural language navigation instruction.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps in the multi-modal UAV intelligent navigation method of any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps in the multi-modal UAV intelligent navigation method of any one of claims 1-7.

Images