Unmanned aerial vehicle autonomous navigation method based on large model and related equipment

Abstract

The invention discloses an unmanned aerial vehicle autonomous navigation method and related equipment based on a large model, and the method comprises the steps: inputting target training data corresponding to an unmanned aerial vehicle task scene and a first visual angle top view image sequence into an initial vision-language model for supervision and fine tuning, and obtaining a candidate vision-language model; based on a composite reward mechanism, performing reinforcement learning on the candidate vision-language model to obtain a target vision-language model; and receiving a current natural language navigation instruction of the user, and generating an unmanned aerial vehicle autonomous navigation prediction result containing a plurality of steps of current action sequences through the target vision-language model according to the current natural language navigation instruction, the current unmanned aerial vehicle state information and the visual memory set. The flight stability and task execution efficiency of the unmanned aerial vehicle in a complex urban environment can be improved, the task success rate is remarkably improved, and the method can be widely applied to the technical field of unmanned aerial vehicle navigation.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) navigation technology, and in particular to an autonomous navigation method and related equipment for UAVs based on a large model. Background Technology

[0002] Currently, the application of unmanned aerial vehicles (UAVs) in urban inspection, emergency rescue, public safety, and infrastructure operation and maintenance is constantly expanding. Their operating environment is gradually evolving from open, regular scenarios to complex, dynamically changing urban spaces. In such environments, UAVs need to continuously make path decisions under limited perception conditions, and their autonomous navigation and intelligent decision-making capabilities have become a crucial technological foundation affecting system safety, efficiency, and scalability. Traditional UAV navigation methods mainly rely on manual remote control or pre-defined route planning based on the Global Positioning System (GPS), which is highly dependent on operators and struggles to flexibly respond to environmental changes and high-level semantic task requirements in complex urban environments. As application scenarios become increasingly diverse, simply relying on coordinates or predefined paths is no longer sufficient to meet actual needs.

[0003] Natural language-based human-computer interaction and vision-and-language navigation (VLN) technologies offer solutions to the aforementioned problems, lowering the operational threshold and improving the adaptability of UAVs to complex scenarios. These technologies are gradually evolving towards data-driven and model-driven approaches. However, current UAV VLN methods based on vision-and-language models (VLMs) still have significant shortcomings: decision-making is mostly based on single-step action prediction, lacking overall planning capabilities and prone to path jitter; insufficient utilization of historical observations and temporal information, resulting in limited spatial memory capabilities; and poor algorithm generalization adaptability, making it difficult to cope with unknown or dynamically changing environments, hindering the application of this technology from theory to real-world UAV scenarios.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The embodiments of this application aim to at least partially address one of the technical problems in the related art. Therefore, the main objective of the embodiments of this application is to propose an autonomous navigation method and related equipment for unmanned aerial vehicles (UAVs) based on a large model. This method can improve the flight stability and mission execution efficiency of UAVs in complex urban environments, significantly increase mission success rate and robustness, thereby enabling UAVs to maintain efficient, stable, and safe navigation capabilities in dynamic or unknown environments, reducing the need for human intervention and potential risks.

[0006] To achieve the above objectives, one aspect of this application proposes an autonomous navigation method for unmanned aerial vehicles (UAVs) based on a large model, the method comprising the following steps:

[0007] Multimodal data is collected from UAV mission scenarios; wherein, the multimodal data includes raw training data and a sequence of first-view top-down images corresponding to the UAV, and the raw training data includes natural language navigation commands, UAV status information and UAV action sequences; The original training data is preprocessed to obtain the target training data; The target training data and the first viewpoint top-down image sequence are input into the initial visual-language model for supervised fine-tuning training to obtain a candidate visual-language model; Based on a composite reward mechanism, the candidate visual-language model is trained by reinforcement learning to obtain the target visual-language model, and the target visual-language model that has passed model verification is deployed to the target UAV control terminal. The system receives the current natural language navigation command input by the user, and generates an autonomous navigation prediction result for the UAV containing a sequence of several current actions based on the target visual-language model in the target UAV control terminal, the current natural language navigation command, the current UAV state information, and the visual memory set; wherein, the visual memory set is constructed from the historical first-person top-view image sequence based on the progressive interval sampling strategy.

[0008] In some embodiments, after receiving the current natural language navigation command input by the user, and generating an autonomous navigation prediction result for the UAV containing a sequence of several current actions based on the target visual-language model in the target UAV control terminal, according to the current natural language navigation command, current UAV state information, and visual memory set, the method further includes: The current UAV is controlled to execute corresponding flight control operations sequentially according to the current action sequence in the UAV autonomous navigation prediction result; wherein, the length of the current action sequence meets a preset length threshold, and all actions in the current action sequence are within a predefined discrete action space; During the execution of flight control operations, the current UAV status information and current environmental perception information corresponding to the current UAV are dynamically updated; wherein, the current environmental perception information includes a first-view top-down image generated during the execution of flight control operations, the first-view top-down image generated during the execution of flight control operations is used to update the historical first-view top-down image sequence, and the updated historical first-view top-down image sequence is used to update the visual memory set. If there is no stop command in the current action sequence, the process returns to the step of generating a UAV autonomous navigation prediction result containing several steps of the current action sequence based on the target vision-language model in the target UAV control terminal, the current natural language navigation command, the current UAV state information, and the visual memory set, until the stop command is found in the current action sequence, thereby ending the UAV autonomous navigation process.

[0009] In some embodiments, the method further includes the step of constructing the visual memory set based on the progressive interval sampling strategy, wherein constructing the visual memory set based on the progressive interval sampling strategy includes: When the user inputs the current natural language navigation command, a first-view top-down image generated during the current flight of the UAV is collected, and the first-view top-down image is added to the historical first-view top-down image sequence in chronological order. Based on the progressive interval sampling strategy, the time offset sequence corresponding to the historical first-view top-down image sequence is calculated, and the visual memory set is constructed at the current time step according to the time offset sequence; wherein, the time offset sequence is composed of the offset of the target first-view top-down image in the historical first-view top-down image sequence relative to the current time step, and the target first-view top-down image is determined based on a preset historical view sampling rule.

[0010] In some embodiments, the step of preprocessing the original training data to obtain target training data includes: The natural language navigation instructions in the original training data are subjected to data normalization processing to obtain target instruction text data; Based on the reference example trajectory, label data including the UAV state information and the UAV action sequence in the original training data is constructed; wherein, the label data is used for supervised alignment between the action sequence output by the model and the reference trajectory during supervised fine-tuning training, and the label data is also used as input to the sub-target state alignment reward and stopping consistency reward included in the composite reward mechanism during reinforcement learning training; The target training data is determined based on the target instruction text data and the label data.

[0011] In some embodiments, the step of inputting the target training data and the first viewpoint top-down image sequence into the initial vision-language model for supervised fine-tuning training to obtain a candidate vision-language model includes: The target training data and the first view top-down image sequence are input into the initial vision-language model; An autoregressive sequence generation method is used to train the initial visual-language model based on the target training data and the first-view top-down image sequence. The cross-entropy loss function is used as the training objective function to optimize the model parameters of the initial visual-language model, thereby obtaining the candidate visual-language model.

[0012] In some embodiments, the step of training the candidate visual-language model using reinforcement learning based on a composite reward mechanism to obtain a target visual-language model, and then deploying the target visual-language model that has passed model verification to the target UAV control terminal, includes: Construct a multi-objective composite reward function; wherein, the multi-objective composite reward function includes a sub-objective state alignment reward, a stopping consistency reward, and an output format reward, and the sub-objective state alignment reward includes a distance reduction reward and a heading angle reward; At each time step, the sub-reward items in the multi-objective composite reward function are linearly combined according to preset weights to obtain a comprehensive reward value; The candidate visual-language model is trained based on the comprehensive reward value, and the model strategy of the candidate visual-language model is optimized using a group relative strategy optimization method to obtain the target visual-language model.

[0013] To achieve the above objectives, another aspect of this application proposes an autonomous navigation device for unmanned aerial vehicles (UAVs) based on a large model, the device comprising the following modules: A multimodal data acquisition module is used to acquire multimodal data from UAV mission scenarios; wherein, the multimodal data includes raw training data and a sequence of first-view top-down images corresponding to the UAV, and the raw training data includes natural language navigation commands, UAV status information and UAV action sequences; The data preprocessing module is used to preprocess the original training data to obtain the target training data; The supervised fine-tuning training module is used to input the target training data and the first view top-view image sequence into the initial visual-language model for supervised fine-tuning training to obtain a candidate visual-language model; The reinforcement learning training module is used to perform reinforcement learning training on the candidate visual-language model based on a composite reward mechanism to obtain the target visual-language model, and to deploy the target visual-language model that has passed model verification to the target UAV control terminal. The navigation model inference module is used to receive the current natural language navigation command input by the user, and generate an autonomous navigation prediction result for the UAV containing several steps of the current action sequence based on the target vision-language model in the target UAV control terminal, the current natural language navigation command, the current UAV state information, and the visual memory set; wherein, the visual memory set is constructed from the historical first-person top-view image sequence based on the progressive interval sampling strategy.

[0014] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0016] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0017] The embodiments of this application include at least the following beneficial effects: This application provides a UAV autonomous navigation method and related equipment based on a large model. This scheme collects multimodal data from UAV mission scenarios. The multimodal data includes original training data and a sequence of first-view overhead images corresponding to the UAV. The original training data includes natural language navigation commands, UAV state information, and UAV action sequences. The original training data is preprocessed to obtain target training data. The target training data and the first-view overhead image sequences are input into an initial visual-language model for supervised fine-tuning training to obtain a candidate visual-language model. Based on a composite reward mechanism, the candidate visual-language model is trained using reinforcement learning to obtain a target visual-language model. The target visual-language model that has passed model verification is deployed to the target UAV control terminal. The current natural language navigation command input by the user is received. Based on the target visual-language model in the target UAV control terminal, the current natural language navigation command, the current UAV state information, and the visual memory set are used to generate a UAV autonomous navigation prediction result containing several steps of the current action sequence. The visual memory set is constructed from historical first-view overhead image sequences based on a progressive interval sampling strategy. This application employs a multi-step action sequence prediction mechanism to generate several future actions in each round of decision-making, achieving phased planning and reducing error accumulation caused by frequent perception-decision loops in traditional single-step control methods. This improves the flight stability and mission execution efficiency of UAVs in complex urban environments. Based on a progressive interval sampling strategy, a visual memory set is constructed from historical first-person overhead image sequences. This significantly reduces visual input redundancy and computational overhead while preserving key temporal and spatial information, providing efficient support for multimodal reasoning and action decision-making, and improving the overall system response speed and reasoning efficiency. By designing a multi-objective composite reward mechanism for UAV navigation tasks, the strategy is comprehensively evaluated from multiple dimensions, enabling the model to balance navigation efficiency, attitude stability, and action legitimacy during the reinforcement learning phase, significantly improving mission success rate and robustness. Furthermore, this application enables UAVs to maintain efficient, stable, and safe navigation capabilities in dynamic or unknown environments, reducing the need for human intervention and potential risks. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the steps of an autonomous navigation method for unmanned aerial vehicles based on a large model, as provided in an embodiment of this application. Figure 2 This is a schematic diagram of the system architecture of an autonomous navigation method for unmanned aerial vehicles based on a large model, provided in an embodiment of this application. Figure 3 This is a schematic diagram of the algorithm flow for a reinforcement learning stage provided in an embodiment of this application; Figure 4This is a schematic diagram of the algorithm flow of an autonomous navigation method for unmanned aerial vehicles based on a large model, provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an autonomous navigation device for unmanned aerial vehicles based on a large model, provided in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0020] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0021] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0024] (1) Vision-and-Language Navigation (VLN) is a technology that integrates visual perception with natural language commands to drive intelligent agents to complete target localization and path planning.

[0025] (2) Vision-and-Language Model (VLM) is a model that processes images / text simultaneously to form multimodal representations and perform downstream tasks.

[0026] (3) Action space, which is the discrete actions that the UAV can perform, such as ascent, descent, and stop.

[0027] (4) Supervised Fine-Tuning (SFT) uses real labeled data to optimize the model.

[0028] (5) Cross-Entropy Loss: A loss function commonly used in classification tasks to measure the difference between the probability distribution predicted by the model and the true label distribution. For single-sample multi-class classification problems, the cross-entropy loss function... Defined as: ; in, For the number of categories, One-hot encoding of the real label. For the model to the first The predicted probability of a class. In training large language models, cross-entropy loss is used to guide the model to generate outputs consistent with the true answers based on the input, thereby maximizing the match between the prediction and the true distribution.

[0029] (6) Reinforcement Learning (RL) optimizes the model through reward and punishment strategies.

[0030] (7) Relative Policy Optimization (GRPO) is a reinforcement learning algorithm that trains a model by designing a rule-based reward function.

[0031] Currently, the application of unmanned aerial vehicles (UAVs) in urban inspection, emergency rescue, public safety, and infrastructure operation and maintenance is constantly expanding, and their operating environment is gradually evolving from open, regular scenarios to complex, dynamically changing urban spaces. In such environments, UAVs need to continuously make path decisions under limited perception conditions, and their autonomous navigation and intelligent decision-making capabilities have become an important technological foundation affecting system safety, efficiency, and scalability.

[0032] Traditional drone navigation primarily relies on manual remote control or pre-defined route planning based on the Global Positioning System (GPS). While these methods are practical in stable environments with clear mission objectives, they are highly dependent on operators and struggle to adapt to environmental changes and high-level semantic tasks in complex urban environments. As application scenarios become increasingly diverse, simply relying on coordinates or predefined paths is no longer sufficient to meet practical needs. Against this backdrop, natural language-based human-computer interaction has gained attention. By introducing Vision-and-Language Navigation (VLN) technology, drones can directly understand target descriptions and path prompts given by humans in natural language and combine this with visual perception information to make autonomous flight decisions. This approach helps lower the operational threshold, improve mission delivery efficiency, and enhance the flexibility and adaptability of drone systems in complex scenarios. From a technological development perspective, UAV vision-language navigation is gradually evolving from traditional rule-driven methods to data-driven and model-driven approaches. In particular, the development of multimodal learning and large-scale modeling technologies has made it possible to simultaneously model visual information, language semantics, and flight status. Processing multi-source information through a unified model framework is expected to improve UAVs' understanding of complex environments and high-level commands. However, extending vision-language navigation technology from theoretical research to real-world UAV scenarios still requires balancing multimodal information modeling capabilities, navigation decision stability, and the executability and safety of output actions at the algorithmic level. How to achieve efficient and reliable language-guided autonomous navigation for UAVs in complex environments has become an important direction for ongoing research and exploration in this field.

[0033] Although UAV VLN-related methods based on Vision-and-Language Model (VLM) have made some progress in model structure and performance metrics in recent years, from the perspective of algorithm modeling and practical application, the relevant technologies still have the following main shortcomings: (1) Decision modeling is mainly based on single-step actions and lacks the ability to plan in stages and overall: Related methods generally adopt a decision paradigm of predicting a single control action step by step, that is, outputting a discrete action command based only on the current observation at each moment. This single-step decision-making method is difficult to characterize the "continuous operation" or "stage goal" implied in the language command. In complex urban environments, it is easy to have problems such as frequent correction, path jitter or even local optima, making it difficult to achieve efficient and smooth long-distance navigation.

[0034] (2) Insufficient utilization of historical observation and temporal information, and limited spatial memory capacity: In real UAV navigation scenarios, the current perspective often cannot fully reflect the environmental structure, and navigation decisions rely heavily on the memory of historical paths, traversed areas, and key visual cues. However, many current algorithms only use the current image or simply stack historical visual information, lacking an effective screening and organization mechanism for visual temporal information, making it difficult for the model to form stable spatial cognition and path reasoning capabilities.

[0035] (3) Insufficient generalization and adaptability of the algorithm, making it difficult to cope with unknown or changing environments: Some related methods are highly dependent on the training environment or specific scene structure. They can achieve certain results in known environments, but their performance drops significantly when deployed to new urban areas, different layouts, or dynamically changing environments. These methods lack the ability to uniformly model complex semantic combinations and diverse environmental structures at the algorithm level, making it difficult to meet the requirements of robustness and generalization in practical applications.

[0036] In view of this, this application provides a UAV autonomous navigation method and related equipment based on a large model. This scheme employs a multi-step action sequence prediction mechanism to generate several future actions in each round of decision-making, achieving phased planning and reducing error accumulation caused by frequent perception-decision loops in traditional single-step control methods. This improves the flight stability and mission execution efficiency of UAVs in complex urban environments. Based on a progressive interval sampling strategy, a visual memory set is constructed from historical first-person overhead image sequences. This significantly reduces visual input redundancy and computational overhead while preserving key temporal and spatial information, providing efficient support for multimodal reasoning and action decision-making, and improving the overall system response speed and reasoning efficiency. By designing a multi-objective composite reward mechanism for UAV navigation tasks, the strategy is comprehensively evaluated from multiple dimensions, enabling the model to balance navigation efficiency, attitude stability, and action legitimacy during the reinforcement learning phase, significantly improving mission success rate and robustness. Furthermore, this application enables UAVs to maintain efficient, stable, and safe navigation capabilities in dynamic or unknown environments, reducing the need for human intervention and potential risks.

[0037] This application provides a method for autonomous navigation of unmanned aerial vehicles (UAVs) based on a large model, relating to the field of UAV navigation technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle-mounted terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a method for autonomous navigation of UAVs based on a large model, but is not limited to the above forms.

[0038] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0039] Please see Figure 1 , Figure 1 This is an optional flowchart of a UAV autonomous navigation method based on a large model, provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0040] Step S101: Collect multimodal data from the UAV mission scenario; wherein, the multimodal data includes raw training data and a sequence of first-view top-down images corresponding to the UAV, and the raw training data includes natural language navigation commands, UAV status information and UAV action sequences; Optionally, the UAV mission scenario refers to a specific environment pre-planned or constructed for UAVs to conduct flight activities in order to train the model. This is a pre-simulation scenario for data collection, distinct from the scenario in which the model actually performs the task. It is understood that the UAV mission scenario in this step can be a flight area that perfectly matches the characteristics of the actual operating environment, a similar alternative scenario, or a virtual scenario generated through simulation, etc. This application embodiment does not impose any limitations on this; its core purpose is to collect multimodal data that meets the model training requirements for subsequent model training.

[0041] Multimodal data is used for model training and includes raw training data and a sequence of first-person view (FAD) images corresponding to the UAV. The raw training data includes natural language navigation commands, UAV status information, and UAV action sequences. Natural language navigation commands are commands given in natural language to guide the UAV's flight. UAV status information may include, but is not limited to, data reflecting the UAV's current operating status, such as its position, speed, and attitude. UAV action sequences may include, but are not limited to, a series of actions performed by the UAV, such as forward movement, turning, and ascent / descend. The FAD image sequence refers to a set of images taken by the UAV from a top-down perspective, which is used as historical views in model training.

[0042] Step S102: Perform data preprocessing on the original training data to obtain target training data; In some embodiments, step S102 may include: performing data normalization processing on the natural language navigation instructions in the original training data to obtain target instruction text data; constructing label data including UAV state information and UAV action sequences based on the reference example trajectory and according to the UAV state information and UAV action sequences in the original training data; wherein, the label data is used for supervised alignment between the action sequences output by the model and the reference trajectory during supervised fine-tuning training, and the label data is also used as input to the sub-target state alignment reward and stopping consistency reward included in the composite reward mechanism during reinforcement learning training; and determining target training data based on the target instruction text data and the label data.

[0043] The reference example trajectory refers to the standard trajectory used to match the state information of the UAV at each time step with a reference action sequence and to construct label data. Its type can be selected according to the actual situation. The reference example trajectory can be an expert example trajectory, which refers to the UAV flight trajectory with optimal execution, pre-planned by UAV technical experts in combination with the target operation scenario, navigation task requirements, and UAV hardware characteristics, and serves as a reference for constructing label data.

[0044] Optionally, the target training data includes target instruction text data and label data, and the first-person top-down image sequence will participate in model training as a historical view during the model training phase.

[0045] In the specific implementation, the process of normalizing the natural language navigation instructions in the original training data to obtain the target instruction text data is as follows: Normalization of the natural language navigation instructions includes removing irrelevant symbols, standardizing the semantic expression format, and eliminating obviously ambiguous instructions, thereby ensuring the semantic and structural consistency of the instruction text. The processed target instruction text data serves as the language input for the model training phase, used in both supervised fine-tuning and reinforcement learning stages.

[0046] In the specific implementation, based on the reference example trajectory, the process of constructing labeled data including UAV state information and UAV action sequences from the original training data is as follows: Using UAV state information and UAV action sequences as a foundation, and based on expert example trajectories, labeled data containing UAV state information and reference action sequences is constructed for each time step. The labeled data is used for supervised alignment between the action sequences output by the model and the reference trajectory during supervised fine-tuning training. The labeled data is also used as input to the sub-objective state alignment reward and stopping consistency reward included in the composite reward mechanism during reinforcement learning training.

[0047] Step S103: Input the target training data and the first view top-view image sequence into the initial vision-language model for supervised fine-tuning training to obtain a candidate vision-language model; In some embodiments, step S103 may include: inputting the target training data and the first-view top-down image sequence into the initial visual-language model; using an autoregressive sequence generation method, training the initial visual-language model based on the target training data and the first-view top-down image sequence, and using the cross-entropy loss function as the training objective function to optimize the model parameters of the initial visual-language model to obtain a candidate visual-language model.

[0048] In this embodiment, model training includes two stages: supervised fine-tuning and reinforcement learning. The supervised fine-tuning stage aims to enable the visual-language model to master the basic mapping ability from input (instructions + first-person top-down image sequence) to output (predicted flight action sequence); the reinforcement learning stage aims to optimize the generalization ability and task performance stability of the visual-language model.

[0049] In the first stage of model training, this application employs supervised fine-tuning to train the navigation model end-to-end, guiding the model to establish a stable mapping relationship between multimodal perception inputs and executable action sequences. In this stage, the model's input includes: natural language navigation instructions obtained after data preprocessing, labeled data containing UAV state information and UAV action sequences, and visual input obtained by progressively interval sampling of historical views, where the historical views consist of first-view top-down images (the UAV's top view) acquired in the first-stage data acquisition step. The model's output is a discrete action sequence conforming to predefined action space constraints. Through supervised learning, the model is able to generate action decision results consistent with the reference navigation trajectory under given multimodal context conditions.

[0050] During supervised fine-tuning, the model is trained using an autoregressive sequence generation approach, which involves progressively predicting the action instruction to be executed at the next moment, given the preceding actions. The training objective is modeled as the prediction problem of the next action label. By progressively generating action sequences, the model learns the correspondence between multimodal inputs and action decisions.

[0051] In terms of parameter optimization, the supervised fine-tuning stage uses the cross-entropy loss function as the training objective function to measure the difference between the action sequences generated by the model and the reference action trajectories. By minimizing this loss function, the model parameters are gradually optimized, thereby improving the model's ability to generate correct and executable action sequences under known example conditions.

[0052] Through the above-mentioned supervised fine-tuning process, the model can initially grasp the basic decision-making pattern of UAV vision-language navigation tasks, laying a stable initial foundation for subsequent reinforcement learning to optimize strategies and improve generalization capabilities.

[0053] Step S104: Based on the composite reward mechanism, the candidate visual-language model is trained by reinforcement learning to obtain the target visual-language model, and the target visual-language model that has passed the model verification is deployed to the target UAV control terminal. In some embodiments, step S104 may include: constructing a multi-objective composite reward function; wherein the multi-objective composite reward function includes a sub-objective state alignment reward, a stopping consistency reward, and an output format reward, and the sub-objective state alignment reward includes a distance reduction reward and a heading angle reward; in each time step, the sub-reward items in the multi-objective composite reward function are linearly combined according to preset weights to obtain a comprehensive reward value; the candidate visual-language model is trained based on the comprehensive reward value, and the model strategy of the candidate visual-language model is optimized using a group relative strategy optimization method to obtain the target visual-language model.

[0054] Optionally, the composite reward mechanism refers to a comprehensive reward calculation system that integrates multi-dimensional evaluation indicators during reinforcement learning training, rather than a single-dimensional reward determination method. The composite reward mechanism is embodied by constructing a target composite reward function. This multi-target composite reward function includes sub-target state alignment rewards, stopping consistency rewards, and output format rewards. Sub-target state alignment rewards include distance reduction rewards and heading angle rewards.

[0055] The reward for sub-target state alignment can be understood as follows: during UAV navigation, each intermediate state in the reference trajectory can be considered a sub-target state achieved through a sequence of expert actions, containing reasonable spatial position and heading angle information. One of the goals of the reinforcement learning phase is to guide the model to gradually approach these sub-target states through its own predicted action sequences. To this end, this application designs the reward function from two aspects: positional proximity and heading consistency.

[0056] The distance reduction reward measures the change in distance between the UAV and the corresponding sub-target state before and after executing the predicted action sequence. A positive reward is given when the Euclidean distance between the UAV and the sub-target decreases after executing the action sequence compared to the current moment; no reward is given if the distance does not decrease. This design guides the model to generate action sequences that help shorten the navigation path and gradually approach the target area.

[0057] The heading angle reward measures whether the heading angle of the UAV after executing a sequence of actions is consistent with the expected heading of the sub-target state. By calculating the difference in heading angles between the two, a positive reward is given when the difference is within a preset tolerance range, thereby guiding the model to optimize spatial position while taking into account the rationality and continuity of flight attitude.

[0058] Regarding the stop consistency reward, it can be understood that in visual-language navigation tasks, the model's prediction of the "stop" action directly determines the timing of the navigation process termination. Incorrect stop decisions may cause the UAV to terminate the mission prematurely or continue performing invalid actions when it should have ended, thus affecting the overall navigation success rate. To improve the reliability of the model's termination judgment, this application designs a stop consistency reward mechanism: a positive reward is given when the action sequence predicted by the model is consistent with the reference trajectory in terms of termination behavior; no reward is given when there is inconsistency between the two in their judgment of whether to terminate. This mechanism can effectively suppress undesirable behaviors such as premature stopping and missed stopping, prompting the model to form a clear and stable termination decision strategy.

[0059] Regarding the output format reward, it is understood that: due to the strict requirements of UAV control systems on the structure and legality of action commands, this application introduces a format reward term in the reinforcement learning stage to ensure the executability of the model's output results. A fixed positive reward is given when the action sequence generated by the model satisfies the predefined action space and output format constraints; otherwise, no reward is given. This reward term is used to constrain the model to maintain the structural correctness and syntactic validity of the output results throughout the policy optimization process.

[0060] At each time step, the system linearly combines the aforementioned sub-reward items according to preset weights to obtain the final comprehensive reward value. By adjusting the weights of different reward items, a flexible balance can be achieved between navigation efficiency, attitude stability, termination accuracy, and output standardization.

[0061] In the policy update process of reinforcement learning, this application uses GRPO to update the model parameters. This method effectively reduces the impact of reward noise on training stability and improves policy convergence efficiency by relatively evaluating multiple candidate action sequences under the same context. Through the above reinforcement learning fine-tuning process, the model, while maintaining the basic navigation capabilities learned in the supervised phase, further improves its generalization ability and decision robustness in complex and unknown environments.

[0062] The target UAV control terminal may include, but is not limited to, the UAV's onboard computing unit and ground station control device.

[0063] After model training is completed, the model is validated on independent test sets and in simulated environments to evaluate its success rate, generalization, and inference interpretability in different task scenarios. If the model does not achieve the expected results, it returns to the training phase for further optimization; otherwise, the validated target vision-language model is saved and deployed to the UAV's onboard computing unit or ground station control device for real-time task execution.

[0064] Step S105: Receive the current natural language navigation command input by the user, and generate an autonomous navigation prediction result for the UAV containing several current action sequences based on the target vision-language model in the target UAV control terminal, according to the current natural language navigation command, the current UAV state information, and the visual memory set; wherein, the visual memory set is constructed from historical first-view overhead image sequences based on a progressive interval sampling strategy.

[0065] In some embodiments, the method further includes a step of constructing a visual memory set based on a progressive interval sampling strategy. This step may include: when receiving a current natural language navigation command input by a user, acquiring a first-view top-down image generated during the current flight of the UAV, and adding the first-view top-down image to a historical first-view top-down image sequence in chronological order; calculating a time offset sequence corresponding to the historical first-view top-down image sequence based on the progressive interval sampling strategy, and constructing a visual memory set at the current time step based on the time offset sequence; wherein the time offset sequence consists of the offset of the target first-view top-down image in the historical first-view top-down image sequence relative to the current time step, and the target first-view top-down image is determined based on a preset historical view sampling rule.

[0066] Here, the current natural language navigation instructions refer to the navigation instructions that the user inputs in real time during the actual execution phase, describing the target location and path-related constraints in natural language text form. It should be noted that the natural language navigation instructions input by the user to the UAV system during the actual execution phase also need to undergo data preprocessing before being used as model input. The data preprocessing process for these natural language navigation instructions is the same as the instruction text data processing process in step S102. A detailed description of the data preprocessing process can be found in the relevant content of the foregoing embodiments, and will not be repeated here.

[0067] The current state information of the UAV refers to the various state parameters of the UAV during real-time flight (such as current position, flight altitude, flight speed, and fuselage attitude angle). It is the basis for the model to generate navigation decisions and is real-time data collected during actual operation, which is different from the UAV state information during the training phase.

[0068] The current action sequence refers to a sequence of consecutive UAV action commands generated by the model for the current navigation requirements. It is a real-time action plan adapted to the current navigation commands, UAV status, and environmental characteristics, distinct from the reference action sequence during the training phase. It represents the directly executable action result output by the model. The length of the predicted action sequence does not exceed a preset upper limit (e.g., eight steps). When the predicted result includes a stop command, the action sequence length can be less than this upper limit. All actions are strictly confined within a predefined discrete action space, including basic control commands such as forward movement, turning, ascent, descent, and stopping, thereby ensuring the structured, executable, and safe nature of the model output.

[0069] In practice, after executing each action sequence in the initial state or during actual flight, the UAV acquires a first-person top-down view image and adds it to the historical view image sequence in chronological order, forming a visual history window for subsequent navigation decisions. It's understandable that in the initial state, the UAV will also directly acquire a first-person top-down view image as its initial input.

[0070] Since directly using complete historical view sequences introduces a large amount of redundant information and significantly increases computational overhead, this application introduces a progressive interval sampling strategy before model inference to construct compact and representative visual memories from historical view image sequences. Specifically, the preset historical view sampling rules based on the progressive interval sampling strategy aim to collect target first-person top-view images from historical view image sequences by starting with the historical view closest to the current time step, tracing back chronologically, and gradually increasing the sampling interval as the tracing depth increases. Then, the offset of the selected target first-person top-view image relative to the current time step is calculated, thereby obtaining a time offset sequence containing multiple offsets. Finally, a visual memory set is constructed based on the time offset sequence, and the visual memory set serves as the input to the model. Through the progressive interval sampling mechanism, the system retains recent key environmental change information while sparsely sampling earlier historical information, thereby effectively reducing the scale of visual input and the computational complexity of inference without significantly losing spatial and contextual information.

[0071] In some embodiments, after step S105, the method may further include: controlling the current UAV to sequentially execute corresponding flight control operations according to the current action sequence in the UAV autonomous navigation prediction result; wherein the length of the current action sequence meets a preset length threshold, and all actions in the current action sequence are within a predefined discrete action space; during the execution of the flight control operation, dynamically updating the current UAV state information and current environmental perception information corresponding to the current UAV; wherein the current environmental perception information includes a first-view top-down image generated during the execution of the flight control operation, the first-view top-down image generated during the execution of the flight control operation is used to update the historical first-view top-down image sequence, and the updated historical first-view top-down image sequence is used to update the visual memory set; if there is no stop command in the current action sequence, the method returns to the step of generating a UAV autonomous navigation prediction result containing several steps of the current action sequence through the target vision-language model in the target UAV control terminal, based on the current natural language navigation command, the current UAV state information, and the visual memory set, until there is a stop command in the current action sequence, thereby ending the UAV autonomous navigation process.

[0072] In the specific implementation, the UAV executes the corresponding flight control operations sequentially according to the current action sequence generated in step S105, and updates its own state information (including but not limited to the UAV's coordinates and orientation angle) and environmental perception information (including but not limited to historical action sequences and first-view top-down images) in real time during the execution process; if there is no stop command in the current action sequence, it returns to the step of generating a UAV autonomous navigation prediction result containing several steps of the current action sequence through the target vision-language model in the target UAV control terminal, based on the current natural language navigation command, the current UAV state information and the visual memory set, until there is a stop command in the current action sequence, so as to end the UAV autonomous navigation process.

[0073] Steps S101 to S105 as illustrated in this embodiment employ a multi-step action sequence prediction mechanism to generate several future actions in each round of decision-making, achieving phased planning and reducing error accumulation caused by frequent perception-decision loops in traditional single-step control methods. This improves the flight stability and mission execution efficiency of UAVs in complex urban environments. A visual memory set is constructed from historical first-person overhead image sequences based on a progressive interval sampling strategy. This significantly reduces visual input redundancy and computational overhead while preserving key temporal and spatial information, providing efficient support for multimodal reasoning and action decision-making, and improving the overall system response speed and reasoning efficiency. By designing a multi-objective composite reward mechanism for UAV navigation tasks, the strategy is comprehensively evaluated from multiple dimensions, enabling the model to balance navigation efficiency, attitude stability, and action legitimacy during the reinforcement learning phase, significantly improving mission success rate and robustness. Furthermore, this embodiment enables UAVs to maintain efficient, stable, and safe navigation capabilities in dynamic or unknown environments, reducing the need for human intervention and potential risks.

[0074] To explain in detail the principles of the technical solution of this application, the overall process of this application will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principles of this application and should not be regarded as a limitation of this application.

[0075] The purpose of the UAV autonomous navigation method based on a large model provided in this application is to: (1) construct a multi-step action prediction mechanism for decision problems of partially observable sequences, so that the UAV can output a continuous and executable action sequence at each decision moment, thereby improving the overall planning ability and execution efficiency of the navigation process and reducing the instability caused by single-step decision; (2) design a progressive interval sampling strategy, by retaining the dense observation of the most recent step and sparsely sampling the observation of the distant step, reducing the size of the historical input without losing key context information, ensuring that the model can fully observe the historical information while also having good efficiency; (3) enhance the structure and controllability of the navigation decision process, so that the model can form a clear and stable decision logic in complex scenarios, reduce the probability of abnormal behavior, improve the reliability and safety of the system in actual UAV applications, and introduce a staged training and optimization strategy, by combining supervised learning and reinforcement learning, jointly optimize the semantic understanding ability and decision strategy of the model, so that it has a stronger generalization ability and adaptability in unknown environments or complex scenarios. Based on this, this application can significantly improve the success rate and reliability of autonomous navigation of UAVs in complex environments, and achieve efficient, robust and interpretable UAV visual-language navigation capabilities.

[0076] In this embodiment, the system mainly consists of three parts: system framework, algorithm flow, and model training. Specifically: the system framework describes the overall operational flow from data acquisition and model training to task execution, reflecting the overall structure and logical loop of the invention; the algorithm flow section elaborates on the task execution stage, explaining how the model receives input signals (including text input and image sampling strategies) and performs multimodal reasoning, path planning, and UAV action execution; the model training section focuses on the construction and optimization of the core model, detailing the training methods and their functions in the two stages of supervised fine-tuning and reinforcement learning.

[0077] Please see Figure 2 , Figure 2 This is a schematic diagram of the system architecture of an autonomous navigation method for unmanned aerial vehicles based on a large model, as provided in an embodiment of this application. Figure 2 As shown in the embodiments of this application, the system framework of an autonomous navigation method for unmanned aerial vehicles based on a large model mainly includes data acquisition, data preprocessing, model training, model validation, model storage, and task execution stages. The specific implementation details of each stage are as follows: (1) First stage, data collection: First, multimodal data is collected from the UAV mission scenario as the basis for subsequent model training. This multimodal data may include, but is not limited to, natural language navigation commands, UAV state information, UAV action sequences, and first-view overhead images (the UAV's top-down view) generated at each time step. The first-view overhead images collected in this step are used as historical views required for the model training phase.

[0078] (2) Second stage, data preprocessing: Before model training, a unified data preprocessing procedure is performed on the raw training data collected in the first stage to ensure the consistency and effectiveness of the training data in the supervised fine-tuning and reinforcement learning stages. The raw training data includes, but is not limited to: natural language navigation commands, UAV status information, and UAV action sequences.

[0079] The specific data preprocessing process includes the following steps: processing instruction text data and constructing and labeling tag data: 1) Command Text Data Processing: The original natural language navigation commands are standardized, including removing irrelevant symbols, unifying semantic expression formats, and eliminating obviously ambiguous commands, thereby ensuring the semantic and structural consistency of the command text. The processed command text serves as the language input for the model training phase, used in both supervised fine-tuning and reinforcement learning.

[0080] 2) Label Data Construction and Annotation: Based on the UAV state information and UAV action sequence collected in step (1), and based on expert example trajectories, label data containing UAV state information and reference action sequences is constructed for each time step. The label data is used for supervised alignment between the model output action sequence and the reference trajectory during the supervised fine-tuning stage, and also serves as the basis for calculating sub-objective state alignment rewards and stopping consistency rewards during the reinforcement learning stage.

[0081] It is understood that the label data in this application embodiment needs to include UAV state information and reference action trajectory. This application does not involve the annotation of image data (first-person top-view images), and the image data is only used as environmental observation in the model training and inference process. However, it should be noted that the first-person top-view images collected during the model training phase are training images pre-collected in the UAV mission scenario; while the first-person top-view images collected during the model inference phase are images collected after the UAV has executed each action sequence in the initialization state or during actual flight.

[0082] (3) The third stage, model training: Model training is divided into two stages: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). Specifically: 1) Supervised fine-tuning: The high-quality labeled data obtained in the second stage is used to perform supervised learning on the visual-language model, so that the model can master the basic mapping ability from input (instructions + first-person top view image sequence) to output (predicted flight action sequence).

[0083] In the first stage of model training, this application employs supervised fine-tuning to train the navigation model end-to-end, guiding the model to establish a stable mapping relationship between multimodal perception inputs and executable action sequences. In this stage, the model's input includes: natural language navigation commands obtained after data preprocessing, UAV state information corresponding to the natural language navigation commands, reference action sequence labels, and visual input obtained by progressively interval sampling of historical views. The historical views consist of first-view top-down images (top view of the UAV) acquired in the first stage data acquisition step. The model's output is a discrete action sequence that conforms to predefined action space constraints. Through supervised learning, the model is able to generate action decision results consistent with the reference navigation trajectory under given multimodal context conditions.

[0084] During supervised fine-tuning, the model is trained using an autoregressive sequence generation approach, which involves progressively predicting the action instruction to be executed at the next moment, given the preceding actions. The training objective is modeled as the prediction problem of the next action label. By progressively generating action sequences, the model learns the correspondence between multimodal inputs and action decisions.

[0085] In terms of parameter optimization, the supervised fine-tuning stage uses the cross-entropy loss function as the training objective function to measure the difference between the action sequences generated by the model and the reference action trajectories. By minimizing this loss function, the model parameters are gradually optimized, thereby improving the model's ability to generate correct and executable action sequences under known example conditions.

[0086] Through the above-mentioned supervised fine-tuning process, the model can initially grasp the basic decision-making pattern of UAV vision-language navigation tasks, laying a stable initial foundation for subsequent reinforcement learning to optimize strategies and improve generalization capabilities.

[0087] 2) Reinforcement learning: Based on supervised fine-tuning, a composite reward mechanism (including target accuracy reward, reasoning rationality reward, and format standardization reward) is further introduced to optimize the model's generalization ability and task execution stability.

[0088] Please see Figure 3 , Figure 3 This is a schematic diagram of an algorithm flow for the reinforcement learning stage provided in an embodiment of this application, such as... Figure 3 As shown, the specific implementation details of the reinforcement learning phase are as follows: After completing supervised fine-tuning, this application further introduces a reinforcement learning fine-tuning stage to improve the decision-making quality, stability, and accuracy of termination judgment of the navigation strategy in complex environments. In this stage, the model's input is consistent with that of the supervised fine-tuning stage, including: natural language navigation instructions obtained after data preprocessing, UAV state information corresponding to the natural language navigation instructions, reference action sequence labels, and visual input obtained by progressively interval sampling from historical views.

[0089] In view of the characteristics of UAV vision-language navigation tasks, this application designs a multi-objective composite reward function to constrain and guide navigation behavior from multiple perspectives, and uses the Group Relative Policy Optimization (GRPO) method to optimize the model policy.

[0090] The reward function in the reinforcement learning phase consists of multiple complementary sub-reward items, used to comprehensively evaluate the model's behavior from different perspectives, including position, attitude, termination decision, and output format. Each sub-reward item is calculated at each time step based on the execution result of the action sequence predicted by the model. The multi-objective composite reward function specifically includes sub-objective state alignment rewards (distance reduction reward and heading angle reward), stopping consistency rewards, and output format rewards, specifically: A. Sub-target state alignment reward: In UAV navigation, each intermediate state in the reference trajectory can be considered a sub-target state achieved through a sequence of expert actions, containing reasonable spatial position and heading angle information. One of the goals of the reinforcement learning phase is to guide the model to gradually approach these sub-target states through its own predicted action sequences. To this end, this application designs the reward function from two aspects: positional proximity and heading consistency.

[0091] a1. Distance Reduction Reward: This reward measures the change in distance between the UAV and the corresponding sub-target state before and after executing the predicted action sequence. A positive reward is given when the Euclidean distance between the UAV and the sub-target decreases after executing the action sequence compared to the current moment; no reward is given if the distance does not decrease. This design guides the model to generate action sequences that help shorten the navigation path and gradually approach the target area.

[0092] a2. Heading Angle Reward: This reward measures whether the heading angle of the UAV after executing a sequence of actions is consistent with the expected heading of the sub-target state. By calculating the difference in heading angle between the two, a positive reward is given when the difference is within a preset tolerance range, thereby guiding the model to optimize spatial position while taking into account the rationality and continuity of flight attitude.

[0093] B. Stop consistency rewards: In vision-language navigation tasks, the model's prediction of the "stop" action directly determines when the navigation process should terminate. Incorrect stop decisions may cause the UAV to terminate the mission prematurely or continue to perform invalid actions when it should have ended, thus affecting the overall navigation success rate.

[0094] To improve the reliability of the model's termination decisions, this application designs a stopping consistency reward mechanism: a positive reward is given when the action sequence predicted by the model is consistent with the reference trajectory in terms of termination behavior; no reward is given when the two are inconsistent in their determination of whether to terminate. This mechanism can effectively suppress undesirable behaviors such as premature stopping and missed stopping, and promote the model to form a clear and stable termination decision strategy.

[0095] C. Output format bonus: Because UAV control systems have strict requirements for the structure and validity of action commands, this application introduces a format reward term in the reinforcement learning stage to ensure the executability of the model's output. A fixed positive reward is given when the action sequence generated by the model satisfies the predefined action space and output format constraints; otherwise, no reward is given. This reward term is used to constrain the model to maintain the structural correctness and syntactic validity of the output results throughout the policy optimization process.

[0096] At each time step, the system linearly combines the aforementioned sub-reward items according to preset weights to obtain the final comprehensive reward value. By adjusting the weights of different reward items, a flexible balance can be achieved between navigation efficiency, attitude stability, termination accuracy, and output standardization.

[0097] During the policy update process, this application employs GRPO to update the model parameters. This method effectively reduces the impact of reward noise on training stability and improves policy convergence efficiency by relatively evaluating multiple candidate action sequences under the same context. Through the above reinforcement learning fine-tuning process, the model, while maintaining the basic navigation capabilities learned in the supervised phase, further enhances its generalization ability and decision robustness in complex and unknown environments.

[0098] (4) Fourth stage, model validation: After training, the model is validated on independent test sets and in simulated environments to evaluate its success rate, generalization, and inference interpretability in different task scenarios. If the model does not achieve the expected results, it returns to the training phase for further optimization.

[0099] (5) Fifth stage, model saving: The validated model will be saved and deployed to the UAV's onboard computing unit or ground station control device for real-time mission execution.

[0100] (6) Sixth stage, task execution: In practical applications, users input natural language commands, and the model generates inference results and action sequences based on current environmental information. These are then executed by the UAV flight control module to achieve autonomous navigation and mission completion. The inference results serve as the core decision-making basis for the model's generation of action sequences. The action sequences are combinations of continuous / discrete flight control actions that the UAV can directly execute, derived from the inference results and combined with the UAV flight control module's execution specifications, kinematic parameters, and real-time environmental information.

[0101] Please see Figure 4 , Figure 4 This is a schematic diagram of the algorithm flow of an autonomous navigation method for unmanned aerial vehicles based on a large model, provided in an embodiment of this application. Figure 4 As shown, the model's UAV navigation algorithm adopts an iterative closed-loop decision-making mechanism, gradually completing the navigation task through multiple rounds of "perception-memory-reasoning-execution" cycles. Each cycle includes the following steps: 1) Command input: Users input natural language navigation commands into the drone system. These commands describe the target location and path-related constraints in the form of natural language text (e.g., the drone should fly to the left and forward at a T-shaped gray building, which can be set according to the actual situation), serving as high-level semantic input for navigation decisions.

[0102] It should be noted that the natural language navigation commands input by the user into the UAV system during the inference phase also need to be preprocessed before they can be used as input to the model. The data preprocessing process for the natural language navigation commands here is the same as the command text data processing process in the second stage of data preprocessing. For a detailed description of the data preprocessing process, please refer to the relevant content in the foregoing embodiments. This application embodiment will not repeat it here.

[0103] 2) Image Acquisition and Historical Updates: After the UAV completes each action sequence in the initial state or during actual flight, it acquires a first-person top-down view image and adds this first-person top-down view image to the historical view image sequence in chronological order to form a visual history window for subsequent navigation decisions.

[0104] Understandably, in the initial state, the drone will also directly capture a first-person top-down view image as the first input.

[0105] 3) Historical view selection and progressive interval sampling: Since directly using the complete historical view sequence would introduce a large amount of redundant information and significantly increase computational overhead, this application introduces a progressive interval sampling strategy before model inference to construct a compact and representative visual memory from the historical view image sequence.

[0106] Specifically, at the current time step t, the drone has already executed t. One action is performed, and a first-person top-down view image is captured after each action is completed. This results in a sequence of historical view images arranged in chronological order. The historical view image sequence is specifically defined as follows: ; in, This represents the first-view image acquired after the action is completed at the j-th time step (j∈{1,2,...,t-1}). (Images acquired by the UAV in its initial state). To control the input size of the model, at each time step t, the system selects no more than N images from the aforementioned historical views to construct visual memory. Here, N is the preset maximum number of historical view samples; t is the current decision time step, and t≥2 (t=1 is the initial state, no decision is required, and decision-making starts from t=2).

[0107] Next, the historical view sampling process begins with the historical view closest to the current time step, tracing back chronologically, and gradually increasing the sampling interval as the tracing depth increases. Specifically, the offset of the i-th selected first-view top-down image relative to the current time step t is defined as... Offset The calculation formula is as follows: ; in, .

[0108] Based on the time offset sequence containing multiple offsets obtained above Constructing a visual memory set at time step t Visual memory collection Defined as: ; Through the aforementioned progressive interval sampling mechanism, the system retains the most recent key environmental change information while sparsely sampling earlier historical information, thereby effectively reducing the scale of visual input and the complexity of inference computation without significantly losing spatial and contextual information.

[0109] 4) Navigation reasoning and action sequence generation: The model receives natural language navigation commands, current drone state information, and visual memory obtained through progressively spaced sampling. Then, multimodal joint inference is performed to predict a sequence of actions that the drone should perform next.

[0110] The predicted action sequence length does not exceed a preset upper limit (e.g., eight steps). When the prediction result includes a stop command, the action sequence length can be less than this upper limit. All actions are strictly limited to a predefined discrete action space, including basic control commands such as forward, turn, ascend, descend, and stop, thereby ensuring the structured, executable, and safe nature of the model output.

[0111] 5) Action execution: The UAV executes the corresponding flight control operations in sequence according to the action sequence generated in step 4), and updates its own status information (including but not limited to the coordinates and orientation angle of the UAV) and environmental perception information (including but not limited to historical action sequences and first-person top-down view images) in real time during the execution.

[0112] (6) Loop iteration and termination: The above steps are executed repeatedly in a loop until the action sequence generated by the model contains a stop instruction or the preset navigation termination condition is met, at which point the navigation process ends.

[0113] It should be noted that this embodiment is only a brief illustrative description of the overall process of an autonomous navigation method for UAVs based on a large model. Detailed descriptions of each step can be found in the relevant content of the foregoing embodiments, and will not be repeated here. It is understood that this application does not impose any limitations on this.

[0114] In summary, this application addresses the problems of perceptual redundancy, decision instability, and insufficient action execution efficiency in the visual-verbal navigation process of UAVs in complex urban environments. It proposes a navigation algorithm and training optimization scheme for practical flight missions, with key technical points mainly reflected in the following three aspects: (1) A multi-step action sequence prediction mechanism for UAV navigation, wherein the model predicts a multi-step action sequence based on multimodal input in each round of decision-making, rather than a single-step control command, thereby achieving phased navigation decision-making: Unlike related technologies that predict only a single control command at each time step, this application employs a multi-step action sequence prediction mechanism, enabling the model to directly generate a sequence of future actions to be executed in each round of inference. The length of the action sequence generated by the model in this application does not exceed a preset upper limit, and all actions are confined within a predefined discrete action space.

[0115] In this way, the model can perform phased planning at a higher decision-making level, reducing the error accumulation problem caused by frequent perception-decision cycles, thereby improving the flight stability and execution efficiency of UAVs in complex environments.

[0116] (2) A historical visual information modeling method based on progressive interval sampling, that is, using a progressive interval sampling strategy to select historical first-person view images, so as to retain key temporal and spatial information while reducing computational complexity: To address the issues of high redundancy and computational overhead in historical visual information during UAV navigation, this application proposes a progressive interval sampling strategy to construct a compact and representative visual memory from historical first-person view image sequences.

[0117] This progressive interval sampling strategy samples historical views in a "close-to-dense, distant-to-sparse" manner over time. Specifically, it samples recent observations densely and earlier observations sparsely, thus significantly reducing the scale of visual input and the computational complexity of inference while effectively preserving information about environmental evolution and spatial structure. This mechanism provides efficient and stable visual context support for subsequent multimodal inference and action decision-making.

[0118] (3) Multi-objective composite reward mechanism for navigation tasks: To address the multi-dimensional constraints of UAV vision-language navigation tasks, this application designs a multi-objective composite reward mechanism during the reinforcement learning phase to comprehensively evaluate the action sequences generated by the model from multiple perspectives. The reward mechanism includes at least: a positional proximity reward (i.e., distance reduction reward) to guide the UAV to gradually approach the state of the reference sub-target; a heading angle alignment reward to constrain the rationality of the UAV's flight attitude; a stopping consistency reward to improve the accuracy of navigation termination judgment; and an output format reward to ensure the legality of the action sequence structure.

[0119] Through the above-mentioned composite reward design, the model can take into account navigation efficiency, attitude stability, accuracy of termination decision and action executability during the policy optimization process.

[0120] The application scenarios of the UAV autonomous navigation method based on a large model provided in this application include, but are not limited to, UAV autonomous navigation, emergency search and disaster assessment, and urban low-altitude delivery. Compared with current UAV vision-language navigation technologies, this application has significant advantages in the following aspects: (1) Enhancing phased navigation decision-making capabilities: This application generates several future actions in each round of decision-making through a multi-step action sequence prediction mechanism, thereby achieving phased planning. This multi-step action sequence prediction mechanism reduces the error accumulation caused by frequent perception-decision loops in traditional single-step control methods, thereby improving the flight stability and mission execution efficiency of UAVs in complex urban environments.

[0121] (2) Efficient processing of historical visual information: By using a progressive interval sampling strategy to select "near dense and far sparse" historical first-person view images, this application can significantly reduce visual input redundancy and computational overhead while retaining key temporal and spatial information, providing efficient support for multimodal reasoning and action decision-making, and improving the overall response speed and reasoning efficiency of the system.

[0122] (3) Enhance the strategy optimization capability under multi-dimensional constraints: This application designs a multi-objective composite reward mechanism for UAV navigation tasks, which comprehensively evaluates the strategy from multiple dimensions such as sub-objective position proximity, heading angle alignment, termination decision consistency and action output format, so that the model can take into account navigation efficiency, attitude stability and action legality in the reinforcement learning stage, and significantly improve the task success rate and robustness.

[0123] (4) Improve adaptability and safety in complex environments: In view of the problems of perception redundancy, decision instability and insufficient action execution efficiency in UAV navigation in complex urban environments, the algorithm and training optimization scheme of this application can enable UAVs to maintain efficient, stable and safe navigation capabilities in dynamic or unknown environments, reducing the need for manual intervention and potential risks.

[0124] Please see Figure 5 This application also provides an autonomous navigation device 500 for unmanned aerial vehicles based on a large model, which can implement the above-mentioned method. The device includes the following modules: The multimodal data acquisition module 501 is used to acquire multimodal data from the UAV mission scenario; wherein, the multimodal data includes raw training data and a sequence of first-view top-down images corresponding to the UAV, and the raw training data includes natural language navigation instructions, UAV status information and UAV action sequences; Data preprocessing module 502 is used to preprocess the original training data to obtain target training data; The supervised fine-tuning training module 503 is used to input the target training data and the first view top-view image sequence into the initial visual-language model for supervised fine-tuning training to obtain a candidate visual-language model; The reinforcement learning training module 504 is used to perform reinforcement learning training on the candidate visual-language model based on a composite reward mechanism to obtain the target visual-language model, and to deploy the target visual-language model that has passed model verification to the target UAV control terminal. The navigation model inference module 505 is used to receive the current natural language navigation command input by the user, and generate an autonomous navigation prediction result of the UAV containing several steps of the current action sequence based on the target vision-language model in the target UAV control terminal, the current natural language navigation command, the current UAV state information, and the visual memory set; wherein, the visual memory set is constructed from the historical first-person top-view image sequence based on the progressive interval sampling strategy.

[0125] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0126] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0127] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0128] Please see Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 602 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called and executed by the processor 601 using the methods described in the embodiments of this application. The input / output interface 603 is used to implement information input and output; The communication interface 604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 605 transmits information between various components of the device (e.g., processor 601, memory 602, input / output interface 603, and communication interface 604); The processor 601, memory 602, input / output interface 603, and communication interface 604 are connected to each other within the device via bus 605.

[0129] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0130] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0131] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0132] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0133] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0134] This application provides a large-model-based autonomous navigation method and related equipment for unmanned aerial vehicles (UAVs). By employing a multi-step action sequence prediction mechanism, it generates several future actions in each round of decision-making, achieving phased planning and reducing error accumulation caused by frequent perception-decision loops in traditional single-step control methods. This improves the flight stability and mission execution efficiency of UAVs in complex urban environments. Based on a progressive interval sampling strategy, a visual memory set is constructed from historical first-person overhead image sequences. This significantly reduces visual input redundancy and computational overhead while preserving key temporal and spatial information, providing efficient support for multimodal reasoning and action decision-making, and improving the overall system response speed and reasoning efficiency. By designing a multi-objective composite reward mechanism for UAV navigation tasks, the strategy is comprehensively evaluated from multiple dimensions, enabling the model to balance navigation efficiency, attitude stability, and action legitimacy during the reinforcement learning phase, significantly improving mission success rate and robustness. Furthermore, this application enables UAVs to maintain efficient, stable, and safe navigation capabilities in dynamic or unknown environments, reducing the need for human intervention and potential risks.

[0135] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0136] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0139] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0140] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0142] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0144] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for autonomous navigation of unmanned aerial vehicles based on a large model, characterized in that, The method includes the following steps: Multimodal data is collected from UAV mission scenarios; wherein, the multimodal data includes raw training data and a sequence of first-view top-down images corresponding to the UAV, and the raw training data includes natural language navigation commands, UAV status information and UAV action sequences; The original training data is preprocessed to obtain the target training data; The target training data and the first viewpoint top-down image sequence are input into the initial visual-language model for supervised fine-tuning training to obtain a candidate visual-language model; Based on a composite reward mechanism, the candidate visual-language model is trained by reinforcement learning to obtain the target visual-language model, and the target visual-language model that has passed model verification is deployed to the target UAV control terminal. The system receives the current natural language navigation command input by the user, and generates an autonomous navigation prediction result for the UAV containing a sequence of several current actions based on the target visual-language model in the target UAV control terminal, the current natural language navigation command, the current UAV state information, and the visual memory set; wherein, the visual memory set is constructed from the historical first-person top-view image sequence based on the progressive interval sampling strategy.

2. The method according to claim 1, characterized in that, After receiving the current natural language navigation command input by the user, and generating an autonomous navigation prediction result for the UAV containing a sequence of several current actions based on the target visual-language model in the target UAV control terminal, the current natural language navigation command, the current UAV state information, and the visual memory set, the method further includes: The current UAV is controlled to execute corresponding flight control operations sequentially according to the current action sequence in the UAV autonomous navigation prediction result; wherein, the length of the current action sequence meets a preset length threshold, and all actions in the current action sequence are within a predefined discrete action space; During the execution of flight control operations, the current UAV status information and current environmental perception information corresponding to the current UAV are dynamically updated; wherein, the current environmental perception information includes a first-view top-down image generated during the execution of flight control operations, the first-view top-down image generated during the execution of flight control operations is used to update the historical first-view top-down image sequence, and the updated historical first-view top-down image sequence is used to update the visual memory set. If there is no stop command in the current action sequence, the process returns to the step of generating a UAV autonomous navigation prediction result containing several steps of the current action sequence based on the target vision-language model in the target UAV control terminal, the current natural language navigation command, the current UAV state information, and the visual memory set, until the stop command is found in the current action sequence, thereby ending the UAV autonomous navigation process.

3. The method according to claim 1, characterized in that, The method further includes the step of constructing the visual memory set based on the progressive interval sampling strategy, wherein constructing the visual memory set based on the progressive interval sampling strategy includes: When the user inputs the current natural language navigation command, a first-view top-down image generated during the current flight of the UAV is collected, and the first-view top-down image is added to the historical first-view top-down image sequence in chronological order. Based on the progressive interval sampling strategy, the time offset sequence corresponding to the historical first-view top-down image sequence is calculated, and the visual memory set is constructed at the current time step according to the time offset sequence; wherein, the time offset sequence is composed of the offset of the target first-view top-down image in the historical first-view top-down image sequence relative to the current time step, and the target first-view top-down image is determined based on a preset historical view sampling rule.

4. The method according to claim 1, characterized in that, The step of preprocessing the original training data to obtain the target training data includes: The natural language navigation instructions in the original training data are subjected to data normalization processing to obtain target instruction text data; Based on the reference example trajectory, label data including the UAV state information and the UAV action sequence in the original training data is constructed; wherein, the label data is used for supervised alignment between the action sequence output by the model and the reference trajectory during supervised fine-tuning training, and the label data is also used as input to the sub-target state alignment reward and stopping consistency reward included in the composite reward mechanism during reinforcement learning training; The target training data is determined based on the target instruction text data and the label data.

5. The method according to claim 1, characterized in that, The step of inputting the target training data and the first-view top-down image sequence into the initial vision-language model for supervised fine-tuning training to obtain a candidate vision-language model includes: The target training data and the first view top-down image sequence are input into the initial vision-language model; An autoregressive sequence generation method is used to train the initial visual-language model based on the target training data and the first-view top-down image sequence. The cross-entropy loss function is used as the training objective function to optimize the model parameters of the initial visual-language model, thereby obtaining the candidate visual-language model.

6. The method according to claim 1, characterized in that, The step of training the candidate visual-language model using reinforcement learning based on a composite reward mechanism to obtain the target visual-language model, and then deploying the target visual-language model that has passed model verification to the target UAV control terminal, includes: Construct a multi-objective composite reward function; wherein, the multi-objective composite reward function includes a sub-objective state alignment reward, a stopping consistency reward, and an output format reward, and the sub-objective state alignment reward includes a distance reduction reward and a heading angle reward; At each time step, the sub-reward items in the multi-objective composite reward function are linearly combined according to preset weights to obtain a comprehensive reward value; The candidate visual-language model is trained based on the comprehensive reward value, and the model strategy of the candidate visual-language model is optimized using a group relative strategy optimization method to obtain the target visual-language model.

7. An autonomous navigation device for unmanned aerial vehicles based on a large model, characterized in that, The device includes the following modules: A multimodal data acquisition module is used to acquire multimodal data from UAV mission scenarios; wherein, the multimodal data includes raw training data and a sequence of first-view top-down images corresponding to the UAV, and the raw training data includes natural language navigation commands, UAV status information and UAV action sequences; The data preprocessing module is used to preprocess the original training data to obtain the target training data; The supervised fine-tuning training module is used to input the target training data and the first view top-view image sequence into the initial visual-language model for supervised fine-tuning training to obtain a candidate visual-language model; The reinforcement learning training module is used to perform reinforcement learning training on the candidate visual-language model based on a composite reward mechanism to obtain the target visual-language model, and to deploy the target visual-language model that has passed model verification to the target UAV control terminal. The navigation model inference module is used to receive the current natural language navigation command input by the user, and generate an autonomous navigation prediction result for the UAV containing several steps of the current action sequence based on the target vision-language model in the target UAV control terminal, the current natural language navigation command, the current UAV state information, and the visual memory set; wherein, the visual memory set is constructed from the historical first-person top-view image sequence based on the progressive interval sampling strategy.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images