Semantic-based unmanned aerial vehicle autonomous navigation method, device, apparatus and medium

Abstract

The application belongs to the technical field of unmanned aerial vehicle navigation, and relates to a semantic-based unmanned aerial vehicle autonomous navigation method, device, equipment and medium. The method comprises the following steps: acquiring an image sequence and IMU data of an unmanned aerial vehicle, and constructing a three-dimensional geometric skeleton of an environment; forming a two-dimensional semantic map according to the image sequence of the unmanned aerial vehicle; marking corresponding semantic labels according to the three-dimensional geometric skeleton and the two-dimensional semantic map, taking the semantic labels and geometric information as observation results; fusing observation results from different perspectives to obtain a semantic-geometric coupling map; generating different nodes according to the semantic-geometric coupling map, taking the relationship between the nodes as edges connecting the nodes to obtain a dynamic three-dimensional scene graph; acquiring an unmanned aerial vehicle instruction, and generating a flight path according to the dynamic three-dimensional scene graph to realize autonomous navigation of the unmanned aerial vehicle. The application can understand the instruction at the semantic level, thereby realizing autonomous navigation of the unmanned aerial vehicle.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) navigation technology, and in particular to semantic-based UAV autonomous navigation methods, apparatuses, devices, and media. Background Technology

[0002] With the increasing demand for drones in fields such as indoor search and rescue, urban building inspection, and underground space exploration, their ability to fly autonomously in environments where GPS signals cannot be received has become a core technological bottleneck.

[0003] Currently, navigation technologies such as Visual SLAM (Simultaneous Localization and Mapping) and VIO (Visual Inertial Odometry) can, to some extent, solve the problems of "where am I" and "what is the physical structure of the environment" for drones.

[0004] However, the world models constructed by these technologies are essentially purely geometric, containing only spatial information such as points, lines, and surfaces, lacking a deep understanding of the environment's content and functions. This limitation leads to two main problems:

[0005] First, drones cannot understand high-level, abstract mission instructions. Operators cannot give commands in natural language, such as "go to the conference room with the projector on the second floor," but must rely on complex remote control operations or provide precise geographic coordinates.

[0006] Second, drones have limited decision-making capabilities. They cannot use common sense, such as doors being passable and windows being used for lighting, to perform logical reasoning and path planning, resulting in mechanical behavior patterns and a lack of intelligence.

[0007] The main drawback of existing solutions lies in their fundamental cognitive limitations. Technology overlay solutions, due to their fragmented approach of prioritizing geometry over semantics, result in superficial cognition and fail to form a structured understanding of scene relationships; pre-annotation solutions sacrifice autonomy and environmental adaptability, making drones almost unable to operate in unknown or dynamic environments; and reactive tracking solutions lack a global perspective and long-term memory, enabling them to perform only simple, immediate following tasks and failing to complete complex commands requiring multi-step logic. Summary of the Invention

[0008] Therefore, it is necessary to provide a semantic-based method, device, equipment, and medium for autonomous navigation of unmanned aerial vehicles (UAVs) to address the aforementioned technical problems. This method or device can understand instructions at the semantic level, thereby enabling autonomous navigation of UAVs.

[0009] Semantic-based autonomous navigation methods for unmanned aerial vehicles include:

[0010] Acquire image sequences and IMU data from the drone to construct a three-dimensional geometric skeleton of the environment;

[0011] Based on the image sequence of the drone, semantic segmentation is performed on each frame of the image, and a semantic label is assigned to each pixel to form a two-dimensional semantic map;

[0012] Each point of the 3D geometric skeleton is back-projected back to the source 2D image frame, and the corresponding semantic labels are marked according to the 2D semantic map. The semantic labels and geometric information are used as the observation results. As the UAV moves, the same object is observed multiple times. The observation results from different perspectives are fused to obtain a semantic-geometric coupled map.

[0013] Based on the semantic-geometric coupling map, different nodes are generated, and the spatial relationship between nodes and the functional topological relationship between nodes are used as the edges connecting the nodes to obtain a dynamic 3D scene map.

[0014] The system receives instructions from the drone and generates a flight path based on a dynamic 3D scene map to enable autonomous navigation of the drone.

[0015] In one embodiment, it also includes:

[0016] Obtain the correction instructions, parse the correction instructions, and obtain the correction target and correction label;

[0017] Based on the corrected target and corrected labels, update the dynamic 3D scene map and generate a new flight path.

[0018] In one embodiment, it also includes:

[0019] Based on the correction target and correction label, the convolutional neural network for semantic segmentation is trained to learn historical error features, resulting in a well-trained convolutional neural network.

[0020] In one embodiment, it also includes:

[0021] Get the alias of the region in the environment, add the alias attribute to the corresponding node of the region, and update the dynamic 3D scene graph.

[0022] In one embodiment, based on the image sequence of the UAV, semantic segmentation is performed on each frame of the image, and a semantic label is assigned to each pixel to form a two-dimensional semantic map, including:

[0023] Based on the image sequence of the drone, semantic segmentation is performed on each frame of the image, and semantic labels are assigned to each pixel to identify objects. The environment is also divided into regions according to function and category to form a two-dimensional semantic map.

[0024] In one embodiment, based on a semantic-geometric coupled map, different nodes are generated, and edges connecting the nodes are used as the spatial relationships and functional topological relationships between them to obtain a dynamic 3D scene graph, including:

[0025] Based on the semantic-geometric coupling map, spatially adjacent point cloud sets with the same semantic labels are integrated into a node in the scene graph. Each node represents an entity object in the environment and inherits all its semantic and geometric attributes, resulting in different nodes.

[0026] Calculate the distance and relative position between two nodes to obtain the spatial relationship between different nodes, and based on the spatial relationship between different nodes, obtain the functional topological relationship between different nodes;

[0027] A dynamic 3D scene graph is obtained by using the spatial relationships and functional topological relationships between nodes as edges connecting the nodes.

[0028] In one embodiment, obtaining drone commands and generating a flight path based on a dynamic 3D scene map to achieve autonomous drone navigation includes:

[0029] Obtain drone commands and define the start and target nodes;

[0030] Based on the dynamic 3D scene graph, search for the sequence of nodes connected by feasible edges to generate a semantic path;

[0031] The semantic path is decomposed into multiple sub-tasks, and the geometric path of each sub-task is planned to generate a flight path to achieve autonomous navigation of the UAV.

[0032] A semantic-based autonomous navigation device for unmanned aerial vehicles includes:

[0033] The first module is used to acquire image sequences and IMU data from the UAV and construct a three-dimensional geometric skeleton of the environment;

[0034] The second module is used to perform semantic segmentation on each frame of the image based on the image sequence of the UAV, assign semantic labels to each pixel, and form a two-dimensional semantic map.

[0035] The third module is used to back-project each point of the three-dimensional geometric skeleton back to the source two-dimensional image frame, and mark the corresponding semantic labels according to the two-dimensional semantic map, using semantic labels and geometric information as the observation results; as the UAV moves, the same object is observed multiple times, and the observation results from different perspectives are fused to obtain a semantic-geometric coupled map.

[0036] The fourth module is used to generate different nodes based on the semantic-geometric coupling map, and use the spatial relationship and functional topological relationship between nodes as the edges connecting the nodes to obtain a dynamic 3D scene map.

[0037] The fifth module is used to acquire drone commands and generate flight paths based on dynamic 3D scene maps to enable autonomous drone navigation.

[0038] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above method.

[0039] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above method.

[0040] The aforementioned semantic-based UAV autonomous navigation method enables UAVs not only to construct a purely geometric three-dimensional spatial model of the physical world, but also to simultaneously establish a cognitive world model that precisely anchors natural language commands to physical entities, allows for logical reasoning, and continuously evolves through human-machine interaction. Ultimately, this endows UAVs with the ability to truly understand mission intent and environmental functions, transforming their behavior from mechanical point-to-point movement to intelligent task-oriented execution. This bridges the semantic gap between low-level physical perception and high-level semantic cognition, enabling efficient and robust human-machine collaborative operations in complex, unstructured GPS-denied environments, and achieving true human-machine natural collaboration and intelligent autonomous operation. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating a semantic-based autonomous navigation method for unmanned aerial vehicles (UAVs) in one embodiment.

[0042] Figure 2 This is a structural block diagram of a semantic-based autonomous navigation device for unmanned aerial vehicles (UAVs) in one embodiment.

[0043] Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0045] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. In the description of this application, "multiple sets" means at least two sets, such as two sets, three sets, etc., unless otherwise explicitly specified.

[0046] In this application, unless otherwise expressly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two elements or the interaction between two elements, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0047] Furthermore, the technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.

[0048] This application provides a semantic-based autonomous navigation method for unmanned aerial vehicles (UAVs), such as... Figure 1 The flowchart shown, in one embodiment, includes:

[0049] Step 101: Acquire image sequences and IMU data from the UAV to construct a three-dimensional geometric skeleton of the environment.

[0050] Specifically:

[0051] Visual information (such as RGB-D cameras or stereo cameras) is used to acquire image sequences of the drone;

[0052] An inertial measurement unit (IMU) is used to acquire IMU data from the UAV;

[0053] Based on continuous image sequences and IMU data, the six-degree-of-freedom pose (position and attitude) of the UAV in an unknown environment is accurately and incrementally calculated through matching and optimization, and a three-dimensional geometric skeleton of the environment is constructed simultaneously. The specific three-dimensional geometric skeleton is represented as a series of feature points, surfels, or meshes with spatial coordinates.

[0054] In this step, a three-dimensional geometric skeleton of the environment is constructed as a geometric reconstruction flow. It mainly undertakes traditional visual inertial odometry (VIO) or simultaneous localization and mapping (SLAM) tasks and is the foundation for UAVs to understand "where I am" and "what the space looks like".

[0055] It should be noted that how to calculate the six-degree-of-freedom pose of the drone and how to construct the three-dimensional geometric skeleton of the environment are existing technologies and will not be elaborated here.

[0056] Step 102: Based on the image sequence of the UAV, perform semantic segmentation on each frame of the image, assign a semantic label to each pixel, and form a two-dimensional semantic map.

[0057] Specifically:

[0058] Based on the image sequence of the UAV, a lightweight convolutional neural network that runs efficiently on the airborne computing platform is used to perform high-frequency, pixel-by-pixel semantic segmentation on each frame of the image being processed by the VIO system. Semantic labels are assigned to each pixel of the image to identify objects and to divide the environment into regions according to function and category, forming a two-dimensional semantic map.

[0059] Semantic tags include category tags, such as: wall, floor, ceiling, door, window, chair, etc.

[0060] In this step, a two-dimensional semantic map is constructed as a semantic parsing flow.

[0061] It should be noted that lightweight convolutional neural networks and how to perform semantic segmentation are existing technologies and will not be elaborated upon here.

[0062] Step 103: Project each point of the 3D geometric skeleton back to the source 2D image frame, and label the corresponding semantic tags according to the 2D semantic map. Use the semantic tags and geometric information as the observation results. As the UAV moves, the same object is observed multiple times. The observation results from different perspectives are fused to obtain a semantic-geometric coupled map.

[0063] Specifically:

[0064] Using camera parameters, each point of the 3D geometric skeleton is back-projected back to the source 2D image frame (i.e., the corresponding image in the image sequence), and the corresponding semantic label is marked according to the 2D semantic map. The semantic label and geometric information are used as the observation results. The geometric information includes: the 3D coordinates of the point, normal vector, etc.

[0065] As the drone moves, the same object is observed multiple times. The observation results from different perspectives are fused, and a confidence score is calculated for the semantic label of each potential object to obtain a semantic-geometric coupled map. When fusing the observation results, Bayesian update or voting mechanism is used to update the confidence of the semantic label of the object to improve the accuracy and robustness of state estimation (here referring to semantic label), thereby effectively suppressing the noise caused by single-frame recognition errors.

[0066] The confidence score is the probability value of the semantic label with the highest probability in the semantic label probability distribution of an object after multiple observations. It improves the accuracy and robustness of state estimation (here referring to semantic labels). Only stable and reliable semantic labels are used for subsequent scene graph construction and decision-making. How to calculate the confidence score and how to update it using Bayesian and voting mechanisms are existing technologies. Determining the confidence score involves, for example, using a voting mechanism. Assume an object is observed N times during a drone's flight. For each observation, the semantic segmentation network votes for a semantic label (e.g., chair) for the corresponding pixel. After N observations, the label with the most votes is selected as the final label, and its confidence score can be defined as the number of votes for the winning label divided by the total number of observations N. For example, if an object is observed 10 times, identified as a chair 8 times and a table 2 times, then its final label is chair, and the confidence score is 8 / 10 = 80%. Taking Bayesian update as an example, this is a more rigorous probability fusion method that better handles uncertainty. During the process, a probability distribution is maintained for each possible label of each object (chair, table, window, etc.). This initial distribution is called the prior probability. Whenever a new observation (i.e., the segmentation result of a frame) is made, this observation is treated as new evidence. The system uses Bayes' theorem, combining the prior probability and the new evidence, to calculate the updated posterior probability. After multiple observations and updates, the confidence score is the probability value of the label with the highest probability in the object's probability distribution. For example, after multiple updates, if the system considers an object to have a 95% probability of being a door, a 4% probability of being a window, and a 1% probability of being otherwise, then its final label is "door," with a confidence score of 95%.

[0067] In this step, the traditional two-step approach of first building the geometric map and then labeling the semantics is abandoned. The fusion of semantics and geometry occurs at the moment of map element creation. When the geometric reconstruction flow generates a new map point or constructs a small patch in three-dimensional space, the corresponding observation results are generated at the same time.

[0068] Semantic-geometric coupled maps are constructed by tightly coupling two parallel processing streams: geometric reconstruction and semantic parsing. They do not simply perceive the physical form of the environment, but rather generate a world representation that contains both precise geometric structure and rich semantic information in real time and in an integrated manner. This step is a prerequisite for all subsequent advanced cognitive and interactive functions. Its core lies in ensuring the native coupling of semantic and geometric information at the data source, rather than a forced binding through later separate processing. The implementation of this step relies on airborne multimodal sensors, typically a combination of visual information (such as RGB-D cameras or stereo cameras) and an inertial measurement unit (IMU).

[0069] The semantic-geometric coupled map is not a cold, undifferentiated 3D point cloud or grid, but a dynamically updated, semantically deep-annotated, object-oriented 3D map. Every element in this map, whether it's the numerous planes that make up the wall or the sparse set of feature points that make up the chair, carries clear semantic information from its inception. It's like building a digital twin world for a drone, a world that not only reproduces the appearance of the physical space, but more importantly, reveals the function and meaning of each component. This semantically rich geometric model provides the most direct and solid data foundation for the next step—dynamic anchoring and reasoning of scene graphs based on natural language—making the leap from physical space to cognitive space possible.

[0070] It should be noted that Bayesian updates and voting mechanisms are existing technologies and will not be elaborated upon here.

[0071] Step 104: Based on the semantic-geometric coupling map, generate different nodes, and use the spatial relationship between nodes and the functional topological relationship between nodes as the edges connecting the nodes to obtain a dynamic 3D scene map.

[0072] Specifically:

[0073] The semantic-geometric coupled map is instantiated into objects, and point cloud sets with semantic label confidence exceeding a preset threshold (e.g., 75%, where labels are considered stable and reliable) are selected to obtain a preferred point cloud set. A clustering algorithm is used to integrate spatially adjacent preferred point cloud sets with the same semantic labels into a node in the scene graph. This ensures that each node uniquely corresponds to an entity object in the environment, and that only objects repeatedly confirmed by the drone (such as a wall or a fixed table) become official members of the scene graph. Single-frame, sporadic recognition errors (such as misidentifying a reflective surface as a window) are naturally filtered out because they cannot accumulate sufficient confidence over multiple observations, thus ensuring the robustness and accuracy of the scene graph. Each node represents an entity object in the environment and inherits all its semantic attributes (such as the category label "chair" and the color "blue" obtained through visual analysis) and geometric attributes (such as 3D position, size, and orientation), resulting in different nodes.

[0074] Calculate the distance and relative position between the centroids of two nodes to obtain the spatial relationships between different nodes (e.g., near, on top of, under, etc.), and based on the spatial relationships between different nodes, obtain the functional topological relationships between different nodes (e.g., if a node identified as a door is located between two room areas, an edge can be established to connect it and assign it the traversable property).

[0075] In real time and dynamically, various types of relationship edges are calculated and established. The spatial relationship and functional topological relationship between nodes are used as the edges connecting the nodes to obtain a dynamic 3D scene graph.

[0076] In this step, the proximity condition is used to distinguish different entities of the same category. Specifically, proximity determination can employ existing point cloud clustering algorithms to group point clouds belonging to the same physical entity while separating point clouds belonging to different entities. For example, in a conference room, there are multiple chairs, all semantically labeled as "chair." Through spatial proximity determination, point clouds belonging to the first chair are aggregated into one node, and point clouds belonging to the second chair are aggregated into another independent node, instead of incorrectly merging the point clouds of all chairs in the room into one. As for how to use point cloud clustering algorithms to determine proximity, this is a matter of existing technology, such as Euclidean distance-based clustering and density-based clustering.

[0077] Functional topological relationships are inferred by combining the semantic labels of nodes and their spatial relationships, and then applying predefined logical rules. Functional topological relationships include:

[0078] 1) Connection: An entity (usually a passageway) connects two or more areas, such as: a door connecting a corridor and a room, or a staircase connecting the first floor and the second floor;

[0079] 2) Accessible: Mark nodes or connections that drones can traverse, such as corridors, open doors, and open spaces;

[0080] 3) Block: Marks an entity as an impassable physical barrier, such as a wall, a closed door, or a window;

[0081] 4) Separation: An entity forms the boundary between two areas, for example: a wall separates two offices;

[0082] 5) Contains: What objects are contained within a spatial area, defining the hierarchy, for example: a room contains tables and chairs;

[0083] 6) Belongs to / part of: An object is a component of another larger object, for example: a window is part of a wall;

[0084] 7) Support: A flat surface provides physical support for another object, such as a floor supporting a table, or a table supporting a computer;

[0085] 8) Suspended: One object is suspended below another object, for example: a chandelier is suspended from the ceiling;

[0086] 9) Provide access to: Describes functional entrances leading to different floors or specific areas, for example: elevators provide paths to all floors;

[0087] 10) Orientation: The main function of an object is to face another object or direction, for example: a camera faces a door, and a window faces south.

[0088] For example, how to identify node A as a door, node B as a corridor, and node C as an office: Analyze the spatial relationships and find that door A is spatially embedded in a wall separating corridor B and office C. Apply the rule inference function system by using a logical rule: if a 'door' node is located between two accessible areas (such as a room or corridor), then it establishes a connection between the two areas, and this connection is passable.

[0089] In contrast to spatial relationships, functional topological relationships are as follows: Spatial relationships (such as: near, above, below) describe purely geometric locational relationships, answering where something is; while functional topological relationships, such as connecting or suspending, are usually functional or physical relationships, built upon spatial relationships but adding a deeper meaning, answering what something is used for or how it depends on something. For example: a lamp is "below" the ceiling (spatial relationship), but it is also "suspended" from the ceiling (physical dependence); a door is "between" the walls of two rooms (spatial relationship), and this fact allows it to have a "connection" between the two rooms (functional relationship).

[0090] Comparing passage and providing access: they overlap but have different focuses; passage is more like an attribute or label, describing whether an object (such as a door or corridor) itself has the property of being traversed, answering whether it is possible to pass through; providing access focuses more on describing the function or result, because it has the attribute of passage, so it realizes the function of providing access to another area, answering where one can go; generally, passage is the cause / attribute provided, while access is the result / function, and in practical applications they almost coexist.

[0091] This process elevates semantic-geometric coupled maps—a relatively rudimentary form of perceptual data presented in points and surfaces—into a structured, machine-understandable representation of scene knowledge. This step represents a crucial leap from simply seeing objects to understanding a scene, culminating in a dynamic 3D scene graph. This scene graph is not only an abstract representation of the physical world but also the fundamental medium for natural language interaction between the drone and the operator. It precisely and robustly correlates the ambiguity of language with the physical entities of the scene, enabling logical reasoning based on this information.

[0092] The confidence level setting ensures that the foundation (nodes) for constructing the scene graph is reliable, making subsequent path planning more trustworthy. Since the drone will not plan a passage path based on a potentially illusory, low-confidence door, dangerous or ineffective decisions are avoided. Confidence level becomes a crucial quality control checkpoint between raw perception data (step 103) and structured scene knowledge (step 104), ensuring that the drone's cognitive model is built on a stable and reliable observation foundation, which is essential for achieving truly intelligent and safe autonomous navigation.

[0093] The establishment of node relationships (including spatial relationships between nodes and functional topological relationships between nodes) allows the scene description to evolve from "there is a door here" and "there is a corridor there" to "this door is the passage to that corridor".

[0094] When an operator issues a natural language command, such as "go to the red sofa against the corner," the system in this application does not perform a simple keyword search. Instead, it uses a natural language understanding module to parse the command into a target query graph. The query graph contains a core node, "sofa," which must be red and have a proximity (leaning against) relationship with another corner node. Then, in a dynamically constructed 3D scene graph, it searches for a subgraph that matches the structure of this query graph. This process can be called language-based scene graph grounding. Through this graph-based structured matching, the system can pinpoint the unique entity the operator intends to refer to with extreme precision, effectively solving the ambiguity problems caused by repetitive objects and vague commands in complex environments. In addition, the graph-based reasoning capability means that if the command is "find the way to the exit," it will not blindly search on the geometric map but will instead look for a node marked as the exit in the scene graph. Then, it will traverse in reverse the process to find all nodes such as doors and corridors connected by links and walkable edges, thus planning a semantically reasonable escape route at the logical level. This capability enables the drone to understand the deep logic of the task, rather than just the literal meaning of the command.

[0095] The dynamic 3D scene graph is a living, continuously evolving environmental cognition model that transforms the raw perception data of the UAV into a structured knowledge base containing entities, attributes, and relationships. This dynamically updated scene graph provides the UAV with a powerful inference engine that parses natural language commands into specific target objects or a series of task nodes, enabling it to accurately anchor high-level, fuzzy human language to specific targets and behaviors in the physical world.

[0096] It should be noted that clustering algorithms and how to calculate the distance and relative position between the centroids of two nodes are existing technologies and will not be elaborated upon here.

[0097] Step 105: Obtain UAV commands and generate a flight path based on the dynamic 3D scene map to achieve autonomous navigation of the UAV.

[0098] Specifically:

[0099] Obtain drone commands and define the start and target nodes;

[0100] Based on the dynamic 3D scene graph, search for the sequence of nodes connected by feasible edges to generate a semantic path;

[0101] The semantic path is decomposed into multiple sub-tasks, and the geometric path of each sub-task is planned to generate a flight path to achieve autonomous navigation of the UAV.

[0102] In this step, the planning basis for the UAV's autonomous navigation is no longer a simple geometric space, but a dynamic three-dimensional scene graph rich in semantics and relationships. The advanced mission intent (the what and why) is deeply integrated into every aspect of path planning and behavior generation (the how), making the UAV's trajectory not only physically feasible, but also semantically reasonable.

[0103] Traditional navigation planners, such as A or RRT algorithms, essentially search for the shortest or lowest-cost path from a starting point to a destination on a grid or geometric map composed of passable and impassable (obstacle) areas. This approach has a purely physical worldview and fails to grasp the fundamental difference between flying along a corridor and traversing an office in the context of the task. For it, as long as there are no physical collisions, going through an area filled with desks might be a better shortcut. However, this behavior appears strange and illogical to humans.

[0104] This step proposes a semantic navigation planner that uses a 3D scene graph as the main planning space. In this graph, nodes are objects or regions, and edges are the relationships between them. Path planning is no longer about finding a continuous sequence of coordinates in 3D space, but rather about performing a high-level logical path search on the scene graph first.

[0105] For example, for the instruction "Go to Meeting Room 302": First, locate the drone's starting node (the node where it is currently located, such as corridor A) and target node (Meeting Room 3-302) in the scene graph; then, search the graph for a sequence of nodes connected by walkable edges, such as from corridor A to door B, then to corridor C, and finally through door D to enter Meeting Room 3-302. This logical path generated on the graph naturally conforms to the logic of human movement within a building, resulting in a semantic path (corridor-door-corridor-door-room); decompose this macro-planning of the semantic path into a series of specific sub-tasks, and plan the geometric path for each sub-task to generate the flight path. For the subtask of traversing corridor A, the underlying geometry planner will generate a smooth trajectory away from the walls within the semantic region of corridor A. For the subtask of passing through door B, the planner will precisely plan a path through the center of the door frame. The key is that semantic information provides strong constraints and guidance for the underlying geometry planner. The cost function of planning no longer only includes distance or time, but also incorporates the consideration of semantic compliance. For example, flying in areas marked as corridors will have lower costs, while flying in areas marked as office areas but not passageways will incur extremely high costs, thus naturally reflecting respect for the functional zoning of the environment in path selection.

[0106] For complex tasks, such as "inspecting all south-facing windows," the decision planning module first filters out all nodes in the scene graph that satisfy the category (window) and orientation (south) attributes. Then, the system treats it as a high-level Traveling Salesman Problem (TSP) and plans an optimal order and path to access all these target nodes on the topology of the scene graph. When approaching each window node, the system will also automatically execute preset sub-task behaviors, such as adjusting the aircraft's attitude to obtain the best viewing angle perpendicular to the window, taking pictures, or collecting data.

[0107] The flight path generated in this step is a complete, coherent, and intentional trajectory. It breaks down high-level tasks into logically sound steps and translates these steps into safe and efficient flight paths in physical space. The drone's behavior is no longer a simple point-to-point movement, but rather demonstrates an intelligent decision-making process that deeply understands the mission objectives and environmental functions, making its actions more akin to a human assistant with common-sense judgment.

[0108] In this embodiment, the method further includes: obtaining correction instructions, parsing the correction instructions to obtain correction targets and correction labels; updating the dynamic 3D scene map based on the correction targets and correction labels, and generating a new flight path.

[0109] Furthermore, it also includes: training a convolutional neural network for semantic segmentation based on the correction target and correction label to learn historical error features and obtain a trained convolutional neural network.

[0110] Furthermore, it also includes: obtaining aliases for regions in the environment, adding alias attributes to the corresponding nodes of the regions, and updating the dynamic 3D scene graph.

[0111] This embodiment introduces a human-centered continuous learning and correction mechanism. By constructing a continuous human-machine dialogue channel, human experience and common sense are injected into the drone's cognitive core in real time, ensuring the accuracy and timeliness of the drone's world model. This enables the drone to learn from errors, adapt to environmental changes, and continuously accumulate unique knowledge about specific environments. Thus, human-machine interaction is transformed from a traditional one-way command-execution mode into a two-way, co-constructive dialogue-learning mode. This dynamic correction and evolution loop ultimately transforms the drone from a fixed program tool into an intelligent partner that can grow collaboratively with humans and truly adapt to the environment. This gives the drone system the ability to self-improve and evolve knowledge during mission execution.

[0112] Traditional drone systems are often very vulnerable to perception errors or environmental changes. For example, if a system mistakenly identifies a transparent glass door as an open space that can be traversed, it will stubbornly attempt to pass through until it collides or the mission fails. The only solution is usually for the operator to take over control immediately; this interaction is interrupted, and the system's misperception is not fundamentally corrected.

[0113] This application proposes a human-machine-in-the-loop correction mechanism, allowing the operator to intervene at any stage of the task using natural and fluent language. Specifically, in the scenario described above, when the drone plans a path through a glass door, the operator simply issues a voice or text command, "Stop, that's a glass door, you can't go through it." This command is received and parsed by the system's natural language understanding module. The system not only immediately halts the current flight plan but also understands it as a correction command, triggering an online model update process. Using the correction command, the system locates the scene graph node corresponding to "that" (based on the drone's current focus or direction). This node might have been initially marked as an open space or without a specific label due to visual misjudgment. Based on the operator's feedback, the semantic attributes of this node are updated to "glass door," and a new attribute label, such as "obstacle" or "impassable," is added. This update is instantaneous; the topology and attributes of the scene graph are dynamically modified. Subsequently, the decision-making and planning module (the module corresponding to step 105) immediately replans a reasonable path to avoid the glass door based on this updated, more accurate world model, generating a flight path.

[0114] Furthermore, a knowledge evolution process is realized. This data sample, with precise semantic labels and relevant visual features (such as high reflectivity and transparency), is stored in a small knowledge base of the system. Through continuous, limited, and similar interactions, it can perform online, small-sample learning, gradually learning to identify objects that are prone to errors in the initial model. In the future, when encountering similar visual features in the environment, the system can correctly identify them as glass doors with higher confidence, thus achieving an evolution from passive error correction to active prediction.

[0115] Furthermore, knowledge evolution is also reflected in the personalized understanding of the environment. Operators can assign custom aliases to any object or area in the scene graph. For example, they can tell the system to call this corner rest area "Old Wang Corner," and the system will add an alias to the corresponding area node in the scene graph: the attribute of Old Wang Corner. From then on, operators can directly use personalized commands such as "Go to Old Wang Corner," and the system can understand and execute them accurately.

[0116] The aforementioned semantic-based UAV autonomous navigation method enables UAVs not only to construct a purely geometric three-dimensional spatial model of the physical world, but also to simultaneously establish a cognitive world model that precisely anchors natural language commands to physical entities, allows for logical reasoning, and continuously evolves through human-machine interaction. Ultimately, this endows UAVs with the ability to truly understand mission intent and environmental functions, transforming their behavior from mechanical point-to-point movement to intelligent task-oriented execution. This bridges the semantic gap between low-level physical perception and high-level semantic cognition, enabling efficient and robust human-machine collaborative operations in complex, unstructured GPS-denied environments, and achieving true human-machine natural collaboration and intelligent autonomous operation.

[0117] Specifically:

[0118] 1) While generating 3D map elements (such as points and polygons), their semantic attributes are bound in real time and forcibly to construct a 2D semantic map. The semantic and geometric data are designed with a unified data structure. In the VIO / SLAM backend optimization, geometric constraints and semantic consistency are considered at the same time and joint optimization is performed to ensure the native consistency of the world model, which is the foundation of all subsequent cognitive functions.

[0119] 2) Transform the semantic map into an objectified scene graph, automatically generate and reason about multiple types of relationships (spatial, functional, topological), eliminate linguistic ambiguity through graph structure constraints, and thus anchor and reason about the dynamic three-dimensional scene graph based on natural language; parse natural language instructions into query subgraphs and perform real-time matching and reasoning in the constructed dynamic three-dimensional scene graph.

[0120] 3) It receives and understands human correction instructions, updates the correction information to the scene graph nodes and edges in real time, and uses the correction example as a small sample for learning, to carry out online world model correction and evolution, so as to improve the model's future generalization ability, realize knowledge evolution, and form a complete online learning loop of the world model through natural language interaction.

[0121] This application takes semantic understanding as a first principle that runs through the entire process of perception, cognition, decision-making and learning. It belongs to the cutting-edge technology field of deep intersection of robotics and artificial intelligence. Specifically, it focuses on autonomous navigation, semantic scene understanding and natural language human-computer interaction of flying robots (drones) in GPS-denied environments. It integrates artificial intelligence technology, especially scene understanding and natural language processing in computer vision, with traditional robot navigation technology.

[0122] It should be understood that, although Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0123] This application also provides an apparatus, such as Figure 2 As shown, in one embodiment, it includes: a first module 201, a second module 202, a third module 203, a fourth module 204, and a fifth module 205, wherein:

[0124] The first module 201 is used to acquire image sequences and IMU data from the UAV and construct a three-dimensional geometric skeleton of the environment;

[0125] The second module 202 is used to perform semantic segmentation on each frame of the image based on the image sequence of the UAV, assign semantic labels to each pixel, and form a two-dimensional semantic map.

[0126] The third module 203 is used to back-project each point of the three-dimensional geometric skeleton back to the source two-dimensional image frame, and mark the corresponding semantic labels according to the two-dimensional semantic map, using semantic labels and geometric information as the observation results; as the UAV moves, the same object is observed multiple times, and the observation results from different perspectives are fused to obtain a semantic-geometric coupled map.

[0127] The fourth module 204 is used to generate different nodes based on the semantic-geometric coupling map, and use the spatial relationship between nodes and the functional topological relationship between nodes as the edges connecting the nodes to obtain a dynamic three-dimensional scene map.

[0128] The fifth module 205 is used to acquire UAV commands and generate flight paths based on dynamic 3D scene maps to enable autonomous navigation of the UAV.

[0129] For specific limitations regarding semantic-based autonomous navigation devices for unmanned aerial vehicles (UAVs), please refer to the limitations of semantic-based autonomous navigation methods for UAVs mentioned above, which will not be repeated here. Each module in the aforementioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0130] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a semantic-based autonomous navigation method for unmanned aerial vehicles (UAVs). The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0131] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0132] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0133] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0134] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0135] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended application documents.

Claims

1. A semantic-based autonomous navigation method for unmanned aerial vehicles, characterized in that, include: Acquire image sequences and IMU data from the drone to construct a three-dimensional geometric skeleton of the environment; Based on the image sequence of the drone, semantic segmentation is performed on each frame of the image, and a semantic label is assigned to each pixel to form a two-dimensional semantic map; Each point of the 3D geometric skeleton is back-projected back to the source 2D image frame, and the corresponding semantic labels are marked according to the 2D semantic map. The semantic labels and geometric information are used as the observation results. As the drone moves, the same object is observed multiple times. The observations from different perspectives are fused to obtain a semantic-geometric coupled map. Based on the semantic-geometric coupling map, different nodes are generated, and the spatial relationship between nodes and the functional topological relationship between nodes are used as the edges connecting the nodes to obtain a dynamic 3D scene map. It acquires drone commands and generates flight paths based on dynamic 3D scene maps to achieve autonomous drone navigation; Based on the semantic-geometric coupled map, different nodes are generated, and the edges connecting the nodes are used as the spatial relationships and functional topological relationships between them, resulting in a dynamic 3D scene graph, including: Based on the semantic-geometric coupling map, spatially adjacent point cloud sets with the same semantic labels are integrated into a node in the scene graph. Each node represents an entity object in the environment and inherits all its semantic and geometric attributes, resulting in different nodes. Calculate the distance and relative position between two nodes to obtain the spatial relationship between different nodes, and based on the spatial relationship between different nodes, obtain the functional topological relationship between different nodes; A dynamic 3D scene graph is obtained by using the spatial relationships and functional topological relationships between nodes as edges connecting the nodes. 2.The semantic-based autonomous navigation method for UAVs according to claim 1, wherein, Also includes: Obtain the correction instructions, parse the correction instructions, and obtain the correction target and correction label; Based on the corrected target and corrected labels, update the dynamic 3D scene map and generate a new flight path. 3.The semantic-based autonomous navigation method for UAVs according to claim 2, wherein, Also includes: Based on the correction target and correction label, the convolutional neural network for semantic segmentation is trained to learn historical error features, resulting in a well-trained convolutional neural network. 4.The semantic-based autonomous navigation method for UAVs according to claim 3, wherein, Also includes: Get the alias of the region in the environment, add the alias attribute to the corresponding node of the region, and update the dynamic 3D scene graph.

5. The semantic-based autonomous navigation method for UAVs according to any of claims 1 to 4, characterized in that, Based on the image sequence from the drone, semantic segmentation is performed on each frame of the image, and a semantic label is assigned to each pixel to form a two-dimensional semantic map, including: Based on the image sequence of the drone, semantic segmentation is performed on each frame of the image, and semantic labels are assigned to each pixel to identify objects. The environment is also divided into regions according to function and category to form a two-dimensional semantic map.

6. The semantic-based autonomous navigation method for UAVs according to any of claims 1 to 4, characterized in that, Obtain drone commands and generate flight paths based on dynamic 3D scene maps to achieve autonomous drone navigation, including: Obtain drone commands and define the start and target nodes; Based on the dynamic 3D scene graph, search for the sequence of nodes connected by feasible edges to generate a semantic path; The semantic path is decomposed into multiple sub-tasks, and the geometric path of each sub-task is planned to generate a flight path to achieve autonomous navigation of the UAV.

7. A semantic based autonomous navigation apparatus for unmanned aerial vehicles, characterized in that, The semantic-based unmanned aerial vehicle autonomous navigation method according to any one of claims 1 to 6, comprising: a first module configured to acquire an image sequence and IMU data of the unmanned aerial vehicle, and construct a three-dimensional geometric skeleton of an environment; a second module configured to perform semantic segmentation on each frame of the image sequence according to the image sequence of the unmanned aerial vehicle, assign a semantic label to each pixel point, and form a two-dimensional semantic map; a third module configured to project each point of the three-dimensional geometric skeleton back to the original two-dimensional image frame, and mark a corresponding semantic label according to the two-dimensional semantic map, so as to take the semantic label and geometric information as an observation result; as the unmanned aerial vehicle moves, the same object is observed multiple times, and observation results from different perspectives are fused to obtain a semantic-geometric coupled map; a fourth module configured to generate different nodes according to the semantic-geometric coupled map, and take spatial relationships between the nodes and functional topological relationships between the nodes as edges connecting the nodes, to obtain a dynamic three-dimensional scene graph; a fifth module configured to acquire an unmanned aerial vehicle instruction, and generate a flight path according to the dynamic three-dimensional scene graph, to realize autonomous navigation of the unmanned aerial vehicle. 8.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-7. The processor executes the computer program to realize the steps of the method according to any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method according to any one of claims 1 to 6.

Images