Unmanned aerial vehicle target navigation method based on visual language understanding

Abstract

According to the unmanned aerial vehicle target navigation method based on visual language understanding, a problem model is constructed, an instruction coding module, a target retrieval module and a waypoint planning module are established to solve the problem model, and an executable control instruction is output by a PID controller. Through a mechanism of combining natural language instruction coding and image retrieval, the unmanned aerial vehicle can recognize and understand target description containing abstract semantics, semantic perception and instruction execution capability oriented to an open vocabulary set is realized, task generalization is remarkably improved, track points are directly generated from an image sequence, and the task efficiency is improved. And the PID controller outputs an executable control instruction without depending on map construction or a target position label, so that end-to-end closed-loop navigation from sensing to control is realized.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and robotics technology, and in particular relates to a method for drone target navigation based on visual language understanding. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, the demand for robots to perform complex tasks based on natural language commands is increasing daily. In the field of autonomous navigation, this capability not only supports applications such as home assistance, urban inspection, and environmental detection, but also places higher demands on the semantic understanding and autonomous decision-making of intelligent agents. Against this backdrop, visual-language navigation has become a research hotspot. Its core objective is to enable intelligent agents to recognize and navigate target objects efficiently by understanding language commands and perceiving visual information, relying solely on onboard sensors. For example, when given the command "Find a pink toy," the intelligent agent should have the ability to selectively explore based on semantic and visual cues, rather than blindly traversing the entire environment.

[0003] Existing visual navigation technologies mainly fall into three categories: First, traditional methods rely on simultaneous localization and mapping or motion structure reconstruction to achieve localization and navigation through feature point matching and geometric reconstruction, but lack the ability to process semantic commands and cannot achieve deep integration of language and vision; Second, end-to-end learning methods rely on imitation learning or deep reinforcement learning to directly map image inputs to action outputs. Although they show certain performance, they generally suffer from problems such as large training sample requirements, low sampling efficiency, severe overfitting, and significant transfer gap between simulation and reality; Third, modular learning methods introduce learnable modules while maintaining the overall process architecture of traditional methods, possessing a certain degree of generalization and interpretability, but still rely on large-scale real data and are easily affected by the accumulation of module errors, and lack human-like semantic reasoning capabilities.

[0004] In recent years, the emergence of large language models and visual language models has brought new opportunities for the development of visual-language navigation. Large language models can transform natural language instructions into structured semantic prompts, while visual language models can achieve semantic and visual fusion through cross-modal feature alignment. These methods have improved semantic understanding and open-vocabulary target recognition capabilities to some extent. However, existing research mainly focuses on ground-based mobile robots, assuming discrete action spaces and not fully considering the continuous control characteristics of UAVs in three-dimensional space. UAVs also face problems such as frequent changes in perception perspective and unstable visual input during flight, making it difficult to directly transfer existing ground-based visual-language navigation methods to aerial navigation tasks.

[0005] In summary, existing visual-language navigation methods have the following shortcomings in UAV scenarios: (1) the motion modeling method is not compatible with the continuous and complex three-dimensional motion characteristics of UAVs, making it difficult to truly reflect flight behavior; (2) UAVs are easily affected by viewpoint occlusion and dynamic interference during perception, resulting in unstable visual observation; (3) existing methods lack robust understanding and effective generalization ability for open vocabulary targets, which limits their application effect in real complex environments. Summary of the Invention

[0006] In view of this, the present invention aims to overcome the shortcomings of the above-mentioned problems in the prior art and proposes a UAV target navigation method based on visual language understanding.

[0007] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0008] In a first aspect, the present invention provides a UAV target navigation method based on visual language understanding, comprising:

[0009] Constructing a problem model includes: given a high-level language instruction The drone needs to be in the candidate target set The process of retrieving the target image I most semantically relevant to the instruction involves calculating the relationship between the language instruction and each candidate image I. j Cross-modal similarity score between j Completed, the final selected target image is defined as The selected target image T is considered the navigation target of the UAV. During navigation, the UAV needs to avoid a set of obstacles in the environment. Simultaneously, it gradually moves towards the region aligned with T; the above problem is modeled as a partially observable Markov decision process, denoted as... The true state of the environment Intelligent agents cannot be directly observed; For a continuous action space, each action a t =(v t ,ω t (This corresponds to the linear velocity and angular velocity control commands for the UAV.) For the observation space, at time t, the UAV receives input. Each frame Self-centered RGB image captured by an airborne monocular camera, including high-level mission instructions. and sub-target image I g ; Let a be the state transition function, describing the state transition function of the UAV when performing action a. t After the state Transfer to The probability of;

[0010] An instruction encoding module is constructed to convert high-level natural language instructions into structured text prompts, which facilitates subsequent target image retrieval.

[0011] Construct a target retrieval module to retrieve data from a predefined set of candidate targets. In the middle, according to the structured text prompts output by the instruction encoding module Select the target image that is most relevant to the semantics of the prompt;

[0012] A waypoint planning module is constructed to predict a structured navigation path to the target image based on the selected target image and the self-centered observation information currently acquired by the UAV.

[0013] Given a continuous trajectory predicted by the waypoint planning module, where each waypoint is normalized to the interval [-1, 1].

[0014] Furthermore, the implementation process of the instruction encoding module includes:

[0015] Given input instructions It contains l tokens, which the system first converts into a discrete token sequence using a tokenizer T(·). Subsequently, each word x_i is projected into a continuous representation through the embedding matrix E(·). Where d represents the embedding dimension, the resulting embedding sequence {e1,…,e l Input into the pre-trained language model f LLM Generate context-dependent hidden states Based on this, the model autoregressively generates task-specific prompts. Its generation process is modeled by conditional distribution:

[0016]

[0017] During the generation process, the system utilizes the decoding strategy π in each generation step. dec Select output from the predicted distribution

[0018] Furthermore, the instruction encoding module also includes introducing a template constraint space. Improving cross-modal semantic alignment and retrieval robustness by limiting the output format.

[0019] Furthermore, the implementation process of the target retrieval module includes:

[0020] Input prompts First, the text is processed by a text encoder to generate text feature vectors. At the same time, each image in the candidate image set The corresponding image feature vector is obtained after processing by the visual encoder. The aforementioned text features and image features achieve cross-modal alignment through contrastive learning; their scaled inner product s is then calculated. j =τ -1 · <t,v j As a similarity score, where τ is a learnable temperature parameter, all similarity scores are then normalized using the softmax function to construct a probability distribution over the candidate set: Then select the image with the highest probability as the target image, that is:

[0021] Furthermore, the specific implementation process of the waypoint planning module includes:

[0022] Given the observation sequence of the most recent P+1 times and target image I g And estimate (1) the time distance to reach the target. (2) A trajectory consisting of H relative waypoints Each waypoint Let represent the egocentric displacement vector at the i-th future step;

[0023] Each frame of image k∈[0,P] will be passed through the visual encoder Independent encoding as feature embedding z t-k =ψ(I t-k These embeddings will be stacked into a matrix.

[0024]

[0025] Define a fusion function φ(·) to fuse the current observations I. t and target image I g The characterization obtained after fusion is This indicates that target-oriented visual cues have been captured;

[0026] This characterization g t The model input is obtained by concatenating the data with the observation matrix.

[0027] X t With position encoding Add them together and input them into a decoder-only array.

[0028] Transformer model

[0029] The final prediction result is obtained through a linear multilayer perceptron:

[0030]

[0031] in, The estimated time and distance to reach the target can be used as a basis for determining whether navigation should stop: if If so, the drone is considered to have reached the target point; otherwise, the predicted waypoint sequence... It will be used to generate the next control action.

[0032] Furthermore, it also includes using a PID controller to convert each waypoint into executable action commands.

[0033] Secondly, the present invention provides an electronic device including a processor and a memory communicatively connected to the processor and used to store executable instructions of the processor, the processor being used to execute the aforementioned UAV target navigation method based on visual language understanding.

[0034] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned UAV target navigation method based on visual language understanding.

[0035] Compared with existing technologies, the UAV target navigation method based on visual language understanding described in this invention has the following advantages:

[0036] This invention enables UAVs to recognize and understand target descriptions containing abstract semantics through a mechanism that combines natural language instruction encoding with image retrieval, achieving semantic perception and instruction execution capabilities for open vocabulary sets, and significantly improving task generalization.

[0037] This invention generates trajectory points directly from image sequences and then outputs executable control commands by a PID controller, achieving end-to-end closed-loop navigation from perception to control without relying on map construction or target location labels.

[0038] This invention achieves target recognition and path planning using images captured solely by an onboard RGB camera without the aid of an external positioning system. It maintains a high success rate even in scenarios with varying lighting, background interference, and dynamic textures, demonstrating its stability and practicality in real-world environments and its ability to adapt to complex real-world conditions.

[0039] Without any fine-tuning, this invention can be directly transferred from the simulation environment to indoor and outdoor physical testing environments, demonstrating excellent cross-domain robustness and policy transfer capabilities, and effectively reducing real-world deployment costs.

[0040] The visual-language navigation framework proposed in this invention can adapt to different types of natural language commands, target image descriptions and environmental complexity, has good task versatility, and can be extended to other robot platforms or autonomous decision-making systems with semantic understanding requirements. It has high theoretical value and engineering application prospects. Attached Figure Description

[0041] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0042] Figure 1 This is a schematic diagram of the overall framework of the method described in this invention;

[0043] Figure 2 A set of simulation environments with progressively increasing complexity for evaluating the performance of the method described in this invention;

[0044] Figure 3 This is a schematic diagram comparing navigation trajectories in simulation using different benchmark methods of the present invention;

[0045] Figure 4 This is a schematic diagram of real indoor flight results of the method described in this invention under different natural language commands;

[0046] Figure 5 Schematic diagram of real outdoor flight results under different natural language commands using the method described in this invention. Detailed Implementation

[0047] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0048] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0049] Example 1

[0050] This invention provides a UAV target navigation method based on visual language understanding, the complete structural framework of which is as follows: Figure 1 As shown, it includes:

[0051] Problem modeling:

[0052] The objective of this invention is to enable unmanned aerial vehicles (UAVs) to achieve autonomous navigation towards visual targets under the guidance of natural language commands. Specifically, given a high-level language command... The drone needs to be in the candidate target set The process of retrieving the target image I most semantically relevant to the instruction involves calculating the relationship between the language instruction and each candidate image I. j Cross-modal similarity score betweenj Completed, the final selected target image is defined as The selected target image T is considered the navigation target of the UAV. During navigation, the UAV needs to avoid a set of obstacles in the environment. Simultaneously, it gradually moves towards the region aligned with T. Since the UAV can only obtain local observation information, the entire mission relies on partially observable sensory inputs and semantic alignment results. To model the above process, this invention models it as a partially observable Markov decision process, denoted as... The true state of the environment Intelligent agents cannot be directly observed; For a continuous action space, each action a t =(v t ,ω t (This corresponds to the linear velocity and angular velocity control commands for the UAV.) For the observation space, at time t, the UAV receives input. Each frame Self-centered RGB image captured by an airborne monocular camera, including high-level mission instructions. and sub-target image I g ; Let a be the state transition function, describing the state transition function of the UAV when performing action a. t After the state Transfer to The probability of this is determined. Throughout the process, the UAV must rely on partial observations and semantic alignment results to complete navigation. Unlike existing ground-based visual navigation methods that mostly employ discrete sets of actions (such as fixed-step translation or fixed-angle rotation), this invention uses a continuous velocity control method, which is more consistent with the real flight characteristics of UAVs in three-dimensional space and can effectively model complex spatial maneuvering behaviors.

[0053] This invention first constructs an instruction encoding module to convert high-level natural language instructions into structured text prompts, facilitating subsequent target image retrieval. Specifically, given an input instruction... It contains l tokens, which the system first converts into a discrete token sequence using a tokenizer T(·). Subsequently, each word x_i is projected into a continuous representation through the embedding matrix E(·). Where d represents the embedding dimension. The resulting embedding sequence {e1,…,e...} l Input into the pre-trained language model f LLM Generate context-dependent hidden states

[0054]

[0055] Based on this, the model autoregressively generates task-specific prompts. Its generation process is modeled by conditional distribution:

[0056]

[0057] During the generation process, the system utilizes the decoding strategy π in each generation step. dec Select output from the predicted distribution To reduce generation ambiguity and ensure the standardization of prompt format, this invention further introduces a template constraint space. For example, "...a photograph," by limiting the output format, improves cross-modal semantic alignment and retrieval robustness. After processing by this module, the final standardized text prompt is obtained. It will be passed as input to the target retrieval module to select the target image that is most relevant to the semantics of the instruction.

[0058] This invention constructs a target retrieval module for searching a predefined set of candidate images. In the middle, according to the structured prompts output by the instruction encoding module The target image most relevant to the semantics of the prompt is selected. To achieve cross-modal information alignment and semantic matching, this module introduces a pre-trained visual language model, mapping natural language and images simultaneously to a unified feature space. This enables the comparison and matching between high-level language and visual semantics. Specifically, it involves input prompts. First, the text is processed by a text encoder to generate text feature vectors. At the same time, each image in the candidate image set The corresponding image feature vector is obtained after processing by the visual encoder. The aforementioned text and image features achieve cross-modal alignment through contrastive learning. To measure the semantic relevance between the input instruction and the candidate image, this invention chooses to calculate their scaled inner product s. j =τ -1 · <t,v j The similarity score is used as a benchmark, where τ is a learnable temperature parameter used to adjust the sensitivity of the similarity distribution. A higher similarity score indicates a semantically closer image to the current prompt. Subsequently, this module normalizes all similarity scores using a softmax function to construct a probability distribution over the candidate set. Then, the image with the highest probability is selected as the target image, that is: Through the aforementioned retrieval mechanism, this module achieves semantic alignment from high-level natural language instructions to specific image targets, accurately identifying the visual target image pointed to by the user even in the absence of explicit category labels. The final selected target image I... gThis information, along with the self-centered observation information currently acquired by the UAV, will be used in the subsequent waypoint generation process to guide it in completing its navigation task.

[0059] This invention constructs a waypoint planning module, which predicts a structured navigation path to a target image based on time-stacked egocentric visual observations. Specifically, given the observation sequence of the most recent P+1 time steps... and target image I g This module will estimate: (1) the time and distance to reach the target. (2) A trajectory consisting of H relative waypoints Each waypoint This represents the egocentric displacement vector for the i-th future step. To extract spatial and temporal features, each frame of the image... k∈[0,P] will be passed through the visual encoder Independent encoding as feature embedding z t-k =ψ(I t-k These embeddings will be stacked into a matrix. To incorporate target condition information, we define a fusion function φ(·) to fuse the current observation I. t and target image I g The characterization obtained after fusion is as follows: This representation captures target-oriented visual cues. Next, this module will use this representation g... t The model input is obtained by concatenating the data with the observation matrix. This module further integrates X t With position encoding They are added together and then fed into a Transformer model that contains only a decoder. The final prediction result is obtained through a linear multilayer perceptron: in, The estimated time and distance to reach the target can be used as a basis for determining whether navigation should stop: if If so, the drone is considered to have reached the target point; otherwise, the predicted waypoint sequence... This module will be used to generate the next control action. It is designed to ensure that path prediction is not only temporally continuous but also responsive to semantic objectives, thus supporting structured control in partially observable, visually complex environments.

[0060] Finally, this invention provides a continuous trajectory predicted by the waypoint planning module, where each waypoint is normalized to the interval [-1, 1]. This invention employs a simple PID controller to convert each waypoint into an executable action command. Specifically, each predicted waypoint... The two components are first scaled linearly: Where v max f represents the maximum permissible linear velocity. c To control the frequency, the inversely normalized displacement vector is then input into the PID controller, mapped to continuous linear and angular velocities. These high-level motion commands will be executed by the UAV in real time.

[0061] The effectiveness of the method of the present invention will be verified through simulation experiments below.

[0062] The visual-language navigation method proposed in this invention has been validated in the Unity simulation platform. To evaluate the generalization ability and robustness of the proposed method under different complexity environments, this invention designed three representative simulation test scenarios: a simple box-shaped obstacle environment, a medium-complexity environment containing everyday furniture such as tables and chairs, and a highly complex and chaotic environment containing numerous irregular obstacles, as shown in the attached diagram. Figure 2 As shown in the diagram. In the simulation, the UAV, relying solely on first-person view image input, needs to complete an obstacle avoidance navigation task from the starting point to the target point. It is worth noting that all methods were trained only in a box obstacle environment; furniture and irregular obstacle scenes were introduced for the first time during the testing phase to verify the model's generalization performance in unseen environments.

[0063] To comprehensively compare the performance of the proposed method, five representative navigation benchmark methods were selected for comparative experiments. The first comparative method is the "Image Input-Based Deep Reinforcement Learning Navigation Method (PPO)," which employs a reinforcement learning paradigm to continuously optimize the policy network through repeated interactions between the agent and the environment. Its input is a monocular image, and its output is continuous action commands, exhibiting typical end-to-end training characteristics. The second method is the "Non-Expert Demonstration and Reinforcement Learning Hybrid Learning Method (NPE-DRL)." This method introduces heuristic demonstration trajectories with a certain bias and combines imitation learning and reinforcement learning to jointly train the policy network, thereby improving the convergence speed and sample utilization efficiency of the policy in the early stages. The third method is the "Image Sequence to Action Sequence Based Recurrent Modeling Method (Seq2Seq)." This method constructs a sequence mapping relationship between images and actions, uses a recurrent encoder-decoder structure to model visual sequences, and outputs action sequences learned from expert demonstrations. The fourth method is the "Cross-Modal Attention Navigation Model (CMA) Integrating Vision and Language." Although this method was originally designed for language-guided navigation tasks, it was adapted for this experiment, retaining only the visual input module and the original decoding structure to evaluate its navigation capabilities in the absence of language input. The fifth method is the "Hybrid APF (Artificial Potential Field-Based Classical Path Planning Method)". This method models the target point as an attraction source and obstacles as repulsion sources, generating a combined force to guide the UAV's movement. It is one of the most widely used algorithms in the traditional path planning field.

[0064] Simulation results are attached. Figure 3 As shown, in moderately and highly complex environments, deep reinforcement learning navigation methods based on image input and those incorporating non-expert policies exhibit significant degradation in unfamiliar environments due to the lack of effective feature alignment mechanisms, resulting in path drift and frequent collisions. While prediction models based on image sequences to action sequences perform stably in the training environment, they are sensitive to changes in spatial structure and have limited generalization performance. Navigation models incorporating image-language cross-modal attention mechanisms also show decreased accuracy and unstable navigation performance in complex environments. Classical path planning methods based on artificial potential fields, while capable of generating reasonable paths in furniture scenes due to their reliance on geometric modeling and manual parameter tuning, often get stuck in local optima in irregular obstacle scenes, leading to navigation failure. In contrast, the method proposed in this invention demonstrates superior navigation performance in multiple test environments. Its generated flight paths are smoother and closer to the shortest path, with virtually no collisions, verifying the robustness and excellent environmental generalization ability of the proposed visual-language guidance framework.

[0065] To verify the generalization ability and application value of the visual-language navigation method described in this invention in real-world environments, the method underwent real-world flight tests in both indoor and outdoor environments, both without any fine-tuning. The indoor test scenario included multiple cubic obstacles with varying surface textures and colors, and different types of target objects such as backpacks, toys, and AprilTags were placed within them. The aircraft used was a Tello Edu model, equipped with a monocular RGB camera with a resolution of 720p, a frame rate of 30 frames per second, and a field of view of 82.6 degrees. The mission required the drone to autonomously navigate to the target object specified by instructions without collision.

[0066] The outdoor environment is more challenging than the indoor environment, with a larger spatial scale, more intense changes in lighting, and complex background interference (such as dense vegetation). The entire navigation process does not rely on any external positioning system (such as OptiTrack), and is entirely based on image input perceived by the camera on the UAV for decision control, thereby verifying the method's ability to understand open-vocabulary targets and cross-domain navigation control.

[0067] The test tasks covered two types of input commands: direct commands (e.g., "fly to the blue backpack") and indirect commands (e.g., "fly to a place where students can put their textbooks"), to comprehensively evaluate the system's ability to understand targets with different semantic complexities. Each command type was tested 20 times in indoor and 20 times in outdoor environments. The results showed that the method achieved a success rate of 83% under direct command conditions and 70% under indirect command conditions. These results demonstrate that the method, without the assistance of an additional perception system, can achieve accurate navigation control of complex semantic targets relying solely on its own image perception, exhibiting good cross-environment transferability and versatility.

[0068] In indirect instruction scenarios, the success rate decreases, primarily due to target matching errors caused by semantic ambiguity. For example, "the place to put textbooks" might apply to multiple target objects (such as backpacks or bookshelves), leading to target inference bias. To further demonstrate the system execution process, see the attached... Figure 4 With appendix Figure 5 The navigation trajectory and waypoint prediction results of the UAV based on the method are shown in indoor and outdoor environments, respectively, demonstrating that the method of the present invention can guide the UAV to avoid obstacles and stably reach the target position under continuous image input.

[0069] Example 2

[0070] An electronic device includes a processor and a memory communicatively connected to the processor and used to store processor-executable instructions, the processor being used to execute the method of Embodiment 1.

[0071] Example 3

[0072] A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor as the method of Embodiment 1 described above.

[0073] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0074] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy and procedures.

[0075] This disclosure is intended to provide implementation schemes for users to selectively prevent the use or access to their personal information data. Specifically, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0076] The acquisition, transmission, storage, use, and processing of data in this disclosed technical solution all comply with the relevant provisions of national laws and regulations.

[0077] It should be noted that in the embodiments disclosed herein, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary and are intended only to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used such solutions.

[0078] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0079] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0080] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0081] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0082] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0083] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0084] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0085] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A UAV target navigation method based on visual language understanding, characterized in that: include: Constructing a problem model includes: given a high-level language instruction The drone needs to be in the candidate target set The process of retrieving the target image I most semantically relevant to the instruction involves calculating the relationship between the language instruction and each candidate image I. j Cross-modal similarity score between j Completed, the final selected target image is defined as The selected target image T is considered the navigation target of the UAV. During navigation, the UAV needs to avoid a set of obstacles in the environment. Simultaneously, it gradually moves towards the region aligned with T; the above problem is modeled as a partially observable Markov decision process, denoted as... The true state of the environment Intelligent agents cannot be directly observed; For a continuous action space, each action a t =(v t ,ω t (This corresponds to the linear velocity and angular velocity control commands for the UAV.) For the observation space, at time t, the UAV receives input. Each frame Self-centered RGB image captured by an airborne monocular camera, including high-level mission instructions. and sub-target image I g ; Let a be the state transition function, describing the state transition function of the UAV when performing action a. t After the state Transfer to The probability of; An instruction encoding module is constructed to convert high-level natural language instructions into structured text prompts, which facilitates subsequent target image retrieval. Construct a target retrieval module to retrieve data from a predefined set of candidate targets. In the middle, according to the structured text prompts output by the instruction encoding module Select the target image that is most relevant to the semantics of the prompt; A waypoint planning module is constructed to predict a structured navigation path to the target image based on the selected target image and the self-centered observation information currently acquired by the UAV. Given a continuous trajectory predicted by the waypoint planning module, where each waypoint is normalized to the interval [-1, 1].

2. The UAV target navigation method based on visual language understanding according to claim 1, characterized in that: The implementation process of the instruction encoding module includes: Given input instructions It contains l tokens, which the system first converts into a discrete token sequence using a tokenizer T(·). Subsequently, each word x_i is projected into a continuous representation through the embedding matrix E(·). Where d represents the embedding dimension, the resulting embedding sequence {e1,…,e l Input into the pre-trained language model f LLM Generate context-dependent hidden states Based on this, the model autoregressively generates task-specific prompts. Its generation process is modeled by conditional distribution: During the generation process, the system utilizes the decoding strategy π in each generation step. dec Select output from the predicted distribution 3. The UAV target navigation method based on visual language understanding according to claim 2, characterized in that: The instruction encoding module also includes the introduction of template constraint space. Improving cross-modal semantic alignment and retrieval robustness by limiting the output format.

4. The UAV target navigation method based on visual language understanding according to claim 2, characterized in that: The implementation process of the target retrieval module includes: Input prompts First, the text is processed by a text encoder to generate text feature vectors. At the same time, each image in the candidate image set The corresponding image feature vector is obtained after processing by the visual encoder. The aforementioned text features and image features achieve cross-modal alignment through contrastive learning; their scaled inner product s is then calculated. j =τ -1 · <t,v j As a similarity score, where τ is a learnable temperature parameter, all similarity scores are then normalized using a softmax function to construct a probability distribution over the candidate set: Then select the image with the highest probability as the target image, that is:

5. The UAV target navigation method based on visual language understanding according to claim 1, characterized in that: The specific implementation process of the waypoint planning module includes: Given the observation sequence of the most recent P+1 times and target image I g And estimate (1) the time distance to reach the target. (2) A trajectory consisting of H relative waypoints Each waypoint Let represent the egocentric displacement vector at the i-th step in the future; Each frame of image k∈[0,P] will be passed through the visual encoder Independent encoding as feature embedding z t-k =ψ(I t-k These embeddings will be stacked into a matrix. Define a fusion function φ(·) to fuse the current observation I. t and target image I g The characterization obtained after fusion is This indicates that target-oriented visual cues have been captured; This characterization g t By concatenating the data with the observation matrix, we obtain the model input. X t With position encoding They are added together and then fed into a Transformer model that contains only a decoder. The final prediction result is obtained through a linear multilayer perceptron: in, The estimated time and distance to reach the target can be used as a basis for determining whether navigation should stop: if If so, the drone is considered to have reached the target point; otherwise, the predicted waypoint sequence... It will be used to generate the next control action.

6. The UAV target navigation method based on visual language understanding according to claim 1, characterized in that: It also includes using a PID controller to convert each waypoint into executable action commands.

7. An electronic device comprising a processor and a memory communicatively connected to the processor and used for storing processor-executable instructions, characterized in that: The processor is used to execute the method described in any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

Images