Unmanned aerial vehicle multi-mode topological map navigation method based on language guidance

Abstract

The invention provides an unmanned aerial vehicle multi-mode topological map navigation method based on language guidance. The unmanned aerial vehicle multi-mode topological map navigation method comprises the steps that a multi-frame image sequence and a natural language task instruction in the flight process of an unmanned aerial vehicle are obtained; based on the natural language task instruction, extracting a semantic key frame from the multi-frame image sequence; based on the semantic key frame, constructing a topological node representing image, language and spatial information; based on the topological node, determining a topological edge of semantic space joint constraint; and on a semantic space topological graph, the optimal path of the unmanned aerial vehicle is determined by adopting a search algorithm, and the semantic space topological graph is constructed by topological nodes and topological edges. According to the invention, the unmanned aerial vehicle can autonomously construct the semantic map in an unknown environment and complete effective navigation according to the language instruction.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a language-guided UAV multimodal topology map navigation method. Background Technology

[0002] With the rapid development of artificial intelligence and intelligent sensing systems, drones are playing an increasingly important role in tasks such as disaster search and rescue, inspection of complex indoor scenes, unmanned delivery, and intelligent patrolling. Traditional drone navigation mainly relies on perception methods such as LiDAR, IMU, and visual SLAM to plan paths by constructing geometric maps. However, existing methods generally have the following shortcomings: Lack of task semantic understanding: Traditional mapping and navigation systems are usually based on low-level perceptual features (such as edges, point cloud density, motion difference) for processing, which makes it difficult to understand "what is the target" and "how to get there" in combination with task instructions. They cannot actively select areas of interest, resulting in navigation target deviation or increased redundant paths.

[0003] Maps lack multimodal semantic information: Current topological maps are mostly constructed using geometric or image features, without integrating high-level information such as language commands and semantic categories, making it difficult for navigation systems to parse and match users' natural language expressions (such as "go to the table next to the kitchen").

[0004] The path planning mechanism does not support semantic reasoning: Most navigation path planning methods (such as A* and Dijkstra) are based only on indicators such as spatial distance or field of view coverage, lacking the participation of language semantics, which makes the system unable to reasonably understand and execute when faced with ambiguous instructions (such as "near the window" or "end of the corridor").

[0005] In summary, traditional UAV navigation exhibits limitations in intelligence and adaptability when faced with challenges such as mission diversity, environmental uncertainty, and complex interactions. Summary of the Invention

[0006] In view of this, this application provides a language-guided UAV multimodal topology map navigation method to solve the above-mentioned technical problems.

[0007] In a first aspect, embodiments of this application provide a language-guided UAV multimodal topology map navigation method, including: Acquire multi-frame image sequences and natural language task commands during the drone's flight process; Based on the natural language task instructions, semantic keyframes are extracted from the multi-frame image sequence; Based on the semantic keyframes, construct topological nodes that represent image, language, and spatial information; Based on the aforementioned topological nodes, determine the topological edges of the semantic space joint constraints; On a semantic space topology graph, a search algorithm is used to determine the optimal path for the UAV, wherein the semantic space topology graph is constructed from topology nodes and topology edges.

[0008] In one possible implementation, semantic keyframes are extracted from the multi-frame image sequence based on the natural language task instructions, including: Multi-frame image sequence Represented as: ;in, For the nth frame image, ; The natural language task instructions are converted into text, and the text is encoded to obtain encoded data. Extract encoded data using the BERT language model. Language task vectors : in, This refers to the encoding function of the BERT language model. Extracting the nth frame image using the CLIP visual coding network Image features : in, For the CLIP visual coding network; This yields the image feature sequence: ; Computational Language Task Vectors Image features of the nth frame semantic similarity between : in, Language task vectors The modulus length; Image features The modulus length; This yields the semantic similarity sequence: ; Calculate the nth frame image semantic changes : judge Is it greater than the threshold? If yes, then the nth frame image is determined to be a semantic keyframe; Otherwise, extract the nth frame image. Cross-modal attention vector ; Calculate the attention change in the nth frame image ; judge If yes, then the nth frame image is determined to be a semantic keyframe; This yields the semantic keyframe sequence: , For the k-th semantic keyframe, , The number of semantic keyframes.

[0009] In one possible implementation, the attention change of the nth frame image is calculated. ;include: or, in, Cross-modal attention vector The Each component.

[0010] In one possible implementation, based on the semantic keyframes, topological nodes representing image, language, and spatial information are constructed; including: Get the semantic keyframes The corresponding UAV state vector ; For the k-th semantic keyframe Image features Projecting yields a first vector of dimension d. : in, The image feature projection matrix; Language task vectors By projecting the language features, we obtain a second vector of dimension d. : in, For language feature projection matrix; For the k-th semantic keyframe The corresponding UAV state vector Projecting yields a third vector of dimension d. : in, The state projection matrix; For the first vector Second vector and the third vector The semantic vector is obtained by fusion. : Then the kth topological node , Natural language semantic labels for topological nodes.

[0011] In one possible implementation, the method further includes: Extract target-class words from natural language task instructions to form a candidate semantic label set, thereby obtaining natural language semantic labels; Alternatively, based on CLIP image-text similarity, the semantic keyframes are matched with the most similar semantic terms in the language tag set, and the matched semantic terms are used as the natural language semantic tags of the nodes.

[0012] In one possible implementation, based on the topological nodes, the topological edges of the semantic space joint constraints are determined; including: Calculate the Euclidean distance between any two topological nodes based on their state vectors: If the Euclidean distance between two topological nodes is less than a preset distance threshold, then the two topological nodes are determined to be physically connected, thereby constructing a first set of basic spatial edges with physical connectivity. : in, For the first Topological nodes For the first Topological nodes for and The Euclidean distance; This is a preset distance threshold; Based on the semantic vectors of the topological nodes, calculate the semantic similarity of any pair of topological nodes in the first basic space edge set. : in, semantic vector The modulus length; semantic vector Modulus length; when Greater than the preset semantic similarity threshold Then, it is determined that the topological node pairs are semantically connected in the task, and a second basic spatial edge set with semantic connectivity is constructed. : Second fundamental space edge set Middle topology node pairs Weight of connected edges for: = in, This is a weighting factor for spatial distance. This is a weighting factor for semantic differences.

[0013] Secondly, embodiments of this application provide a language-guided UAV multimodal topology map navigation device, comprising: The acquisition unit is used to acquire multi-frame image sequences and natural language task commands during the flight of the UAV. The first processing unit is configured to extract semantic keyframes from the multi-frame image sequence based on the natural language task instructions. The second processing unit is used to construct topological nodes representing image, language and spatial information based on the semantic keyframes; The third processing unit is used to determine the topological edges of the semantic space joint constraints based on the topological nodes; The navigation unit is used to determine the optimal path of the UAV on a semantic space topology graph using a search algorithm, wherein the semantic space topology graph is constructed from topology nodes and topology edges.

[0014] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application. Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods of embodiments of this application.

[0015] This application enables drones to autonomously construct semantic maps in unknown environments and complete effective navigation based on language commands. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1A flowchart of a language-guided UAV multimodal topology map navigation method provided in this application embodiment; Figure 2 A functional structure diagram of a language-guided UAV multimodal topology map navigation device provided in this application embodiment; Figure 3 A functional structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0019] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0020] The technical solutions provided in the embodiments of this application will be described below.

[0021] like Figure 1 As shown, this application provides a language-guided UAV multimodal topology map navigation method, including: Step 101: Acquire multi-frame image sequences and natural language task commands during the drone's flight process; For example, the input natural language task instructions are: “ ".

[0022] Step 102: Based on the natural language task instructions, extract semantic keyframes from the multi-frame image sequence; Step 103: Based on the semantic keyframes, construct topological nodes representing image, language, and spatial information; Step 104: Based on the topological nodes, determine the topological edges of the semantic space joint constraints; Step 105: On the semantic space topology graph, a search algorithm is used to determine the optimal path of the UAV, wherein the semantic space topology graph is constructed from topology nodes and topology edges.

[0023] This embodiment provides a novel navigation method that integrates natural language understanding, multimodal perception capabilities, and topological structure representation. It establishes a complete link between "language-perception-map-path," enabling UAVs to autonomously construct semantic maps in unknown environments and complete effective navigation based on language commands. This solves the bottleneck of traditional UAV navigation systems, which exhibit insufficient intelligence and poor adaptability when facing problems such as task diversity, environmental unknownness, and interaction complexity.

[0024] In some embodiments, including: Multi-frame image sequence Represented as: ;in, For the nth frame image, ; The natural language task instructions are converted into text, and the text is encoded to obtain encoded data. Extract encoded data using the BERT language model. Language task vectors : in, This refers to the encoding function of the BERT language model. Extracting the nth frame image using the CLIP visual coding network Image features : in, For the CLIP visual coding network; This yields the image feature sequence: ; Computational Language Task Vectors Image features of the nth frame semantic similarity between : in, Language task vectors The modulus length; Image features The modulus length; This yields the semantic similarity sequence: ; Calculate the nth frame image semantic changes : judge Is it greater than the threshold? If yes, then the nth frame image is determined to be a semantic keyframe; Otherwise, extract the nth frame image. Cross-modal attention vector ; Calculate the attention change in the nth frame image ; judge If so, the nth frame image is considered to have a semantic attention mutation, and the nth frame image is determined to be a semantic keyframe; This yields the semantic keyframe sequence: , For the k-th semantic keyframe, , The number of semantic keyframes.

[0025] This embodiment enhances the semantic sensitivity and model adaptability of keyframe selection, thereby improving the semantic consistency of subsequent map construction and navigation path planning.

[0026] In some embodiments, the attention change of the nth frame image is calculated. ;include: or, in, Cross-modal attention vector The Each component.

[0027] In some embodiments, based on the semantic keyframes, topological nodes representing image, language, and spatial information are constructed, including: Get the semantic keyframes The corresponding UAV state vector ; For the k-th semantic keyframe Image features Projecting yields a first vector of dimension d. : in, The image feature projection matrix; Language task vectors By projecting the language features, we obtain a second vector of dimension d. : in, For language feature projection matrix; For the k-th semantic keyframe The corresponding UAV state vector Projecting yields a third vector of dimension d. : in, The state projection matrix; For the first vector Second vector and the third vector The semantic vector is obtained by fusion. : Then the kth topological node , Natural language semantic labels for topological nodes.

[0028] This embodiment, based on semantic keyframes, further constructs a topological graph node that uniformly represents image, language, and spatial information. This node, as the basic unit of the navigation semantic graph, possesses three capabilities: visual understanding, language alignment, and spatial positioning, significantly enhancing the semantic expressiveness of the graph structure and its navigation decision support capabilities.

[0029] In some embodiments, the method further includes: Extract target-class words from natural language task instructions to form a candidate semantic label set, thereby obtaining natural language semantic labels; Alternatively, based on CLIP image-text similarity, the semantic keyframes are matched with the most similar semantic terms in the language tag set, and the matched semantic terms are used as the natural language semantic tags of the nodes.

[0030] In some embodiments, determining the topological edges of the semantic space joint constraints based on the topological nodes includes: Calculate the Euclidean distance between any two topological nodes based on their state vectors: If the Euclidean distance between two topological nodes is less than a preset distance threshold, then the two topological nodes are determined to be physically connected, thereby constructing a first set of basic spatial edges with physical connectivity. : in, For the first Topological nodes For the first Topological nodes for and The Euclidean distance; This is a preset distance threshold; Based on the semantic vectors of the topological nodes, calculate the semantic similarity of any pair of topological nodes in the first basic space edge set. : in, semantic vector The modulus length; semantic vector Modulus length; when Greater than the preset semantic similarity threshold Then, it is determined that the topological node pairs are semantically connected in the task, and a second basic spatial edge set with semantic connectivity is constructed. : Second fundamental space edge set Middle topology node pairs Weight of connected edges for: = in, This is a weighting factor for spatial distance. This is a weighting factor for semantic differences. It supports adaptive adjustment based on task requirements. It is used to guide cost calculation in path search; a smaller weight indicates a better edge.

[0031] This embodiment ensures the geometric rationality of edges, providing a physical feasibility guarantee for subsequent path execution; it also finds potential path connections based on the semantics of the language task, even if the physical path spans a large distance. Building upon the constructed multimodal topological nodes, a dual constraint mechanism of spatial reachability and semantic relevance is further introduced to achieve effective edge generation in the topological graph and complete path planning oriented towards natural language task objectives; effectively improving the semantic consistency, spatial feasibility, and adaptability to fuzzy expressions in path reasoning.

[0032] Preferably, the search algorithm adopts Graph search algorithm. Let the task language vector be... The endpoint node is determined by semantic matching: Starting node and target node Start and end points, planned route: If the task statement is a vague description (such as "near the window" or "on the left side of the wall"), the system can still flexibly locate the target node by matching language vectors with the node set, thereby realizing the understanding and execution of vague expressions in natural language.

[0033] Based on the above embodiments, this application provides a language-guided UAV multimodal topology map navigation device, see reference. Figure 2 As shown, the language-guided UAV multimodal topology map navigation device 200 provided in this application embodiment includes at least: The acquisition unit 201 is used to acquire multi-frame image sequences and natural language task instructions during the flight of the UAV. The first processing unit 202 is used to extract semantic keyframes from the multi-frame image sequence based on the natural language task instructions. The second processing unit 203 is used to construct topological nodes representing image, language and spatial information based on the semantic keyframes; The third processing unit 204 is used to determine the topological edges of the semantic space joint constraints based on the topological nodes; The navigation unit 205 is used to determine the optimal path of the UAV on a semantic space topology graph using a search algorithm, wherein the semantic space topology graph is constructed from topology nodes and topology edges.

[0034] It should be noted that the principle of the language-guided UAV multimodal topology map navigation device 200 provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the language-guided UAV multimodal topology map navigation device 200 provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0035] Based on the above embodiments, this application also provides an electronic device, see below. Figure 3 As shown, the electronic device 300 provided in this application embodiment includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, it implements the language-guided UAV multimodal topology map navigation method provided in this application embodiment.

[0036] The electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0037] The memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023.

[0038] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0039] Electronic device 300 can also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and with one or more devices that enable a user to interact with electronic device 300 (e.g., mobile phone, computer, etc.), and / or with any device that enables electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 305. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 306. Figure 3 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 3 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0040] It should be noted that, Figure 3 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0041] This application also provides a computer-readable storage medium storing computer instructions. When executed by a processor, these instructions implement the language-guided UAV multimodal topology map navigation method provided in this application. Specifically, the executable program can be built into or installed in an electronic device 300, allowing the electronic device 300 to implement the language-guided UAV multimodal topology map navigation method provided in this application by executing the built-in or installed executable program.

[0042] The language-guided UAV multimodal topology map navigation method provided in this application embodiment can also be implemented as a program product. The program product includes program code. When the program product can run on the electronic device 300, the program code is used to make the electronic device 300 execute the language-guided UAV multimodal topology map navigation method provided in this application embodiment.

[0043] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0044] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on a computing device. However, the program product provided in this application embodiment is not limited thereto. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0045] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0046] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0047] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

Claims

1. A language-guided UAV multimodal topology map navigation method, characterized in that, include: Acquire multi-frame image sequences and natural language task commands during the drone's flight process; Based on the natural language task instructions, semantic keyframes are extracted from the multi-frame image sequence; Based on the semantic keyframes, construct topological nodes that represent image, language, and spatial information; Based on the aforementioned topological nodes, determine the topological edges of the semantic space joint constraints; On a semantic space topology graph, a search algorithm is used to determine the optimal path for the UAV, wherein the semantic space topology graph is constructed from topology nodes and topology edges.

2. The method according to claim 1, characterized in that, Based on the natural language task instructions, semantic keyframes are extracted from the multi-frame image sequence, including: Multi-frame image sequence Represented as: ;in, For the nth frame image, ; The natural language task instructions are converted into text, and the text is encoded to obtain encoded data. Extract encoded data using the BERT language model. Language task vectors : in, This refers to the encoding function of the BERT language model. Extracting the nth frame image using the CLIP visual coding network Image features : in, For the CLIP visual coding network; This yields the image feature sequence: ; Computational Language Task Vectors Image features of the nth frame semantic similarity between : in, Language task vectors The modulus length; Image features The modulus length; This yields the semantic similarity sequence: ; Calculate the nth frame image semantic changes : judge Is it greater than the threshold? If yes, then the nth frame image is determined to be a semantic keyframe; Otherwise, extract the nth frame image. Cross-modal attention vector ; Calculate the attention change in the nth frame image ; judge If yes, then the nth frame image is determined to be a semantic keyframe; This yields the semantic keyframe sequence: , For the k-th semantic keyframe, , The number of semantic keyframes.

3. The method according to claim 2, characterized in that, Calculate the attention change of the nth frame image ; include: or, in, Cross-modal attention vector The Each component.

4. The method according to claim 2, characterized in that, Based on the semantic keyframes, construct topological nodes that represent image, language, and spatial information; include: Get the semantic keyframes The corresponding UAV state vector ; For the k-th semantic keyframe Image features Projecting yields a first vector of dimension d. : in, The image feature projection matrix; Language task vectors By projecting the language features, we obtain a second vector of dimension d. : in, For language feature projection matrix; For the k-th semantic keyframe The corresponding UAV state vector Projecting yields a third vector of dimension d. : in, The state projection matrix; For the first vector Second vector and the third vector The semantic vector is obtained by fusion. : Then the kth topological node , Natural language semantic labels for topological nodes.

5. The method according to claim 4, characterized in that, The method further includes: Extract target-class words from natural language task instructions to form a candidate semantic label set, thereby obtaining natural language semantic labels; Alternatively, based on CLIP image-text similarity, the semantic keyframes are matched with the most similar semantic terms in the language tag set, and the matched semantic terms are used as the natural language semantic tags of the nodes.

6. The method according to claim 4, characterized in that, Based on the topological nodes, determine the topological edges of the semantic space joint constraints; including: Calculate the Euclidean distance between any two topological nodes based on their state vectors: If the Euclidean distance between two topological nodes is less than a preset distance threshold, then the two topological nodes are determined to be physically connected, thereby constructing a first set of basic spatial edges with physical connectivity. : in, For the first Topological nodes For the first Topological nodes for and The Euclidean distance; This is a preset distance threshold; Based on the semantic vectors of the topological nodes, calculate the semantic similarity of any pair of topological nodes in the first basic space edge set. : in, semantic vector The modulus length; semantic vector Modulus length; when Greater than the preset semantic similarity threshold Then, it is determined that the topological node pairs are semantically connected in the task, and a second basic spatial edge set with semantic connectivity is constructed. : Second fundamental space edge set Middle topology node pairs Weight of connected edges for: = in, This is a weighting factor for spatial distance. This is a weighting factor for semantic differences.

7. A language-guided UAV multimodal topology map navigation device, characterized in that, include: The acquisition unit is used to acquire multi-frame image sequences and natural language task commands during the flight of the UAV. The first processing unit is configured to extract semantic keyframes from the multi-frame image sequence based on the natural language task instructions. The second processing unit is used to construct topological nodes representing image, language and spatial information based on the semantic keyframes; The third processing unit is used to determine the topological edges of the semantic space joint constraints based on the topological nodes; The navigation unit is used to determine the optimal path of the UAV on a semantic space topology graph using a search algorithm, wherein the semantic space topology graph is constructed from topology nodes and topology edges.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-6.

Images