Universal unmanned aerial vehicle end-to-end intelligent navigation method based on video multi-mode large model

Abstract

The invention provides a universal unmanned aerial vehicle end-to-end intelligent navigation method based on a video multi-mode large model, and relates to the technical field of unmanned aerial vehicle intelligent navigation, and the method comprises the following steps: S1, text processing: processing an input text through a text encoder, and converting the text into text lexical elements; s2, multi-modal data acquisition: acquiring visual observation information of the unmanned aerial vehicle; and S3, visual observation information extraction: extracting the unmanned aerial vehicle visual observation information key frame through a video encoder, and carrying out denoising, image stabilization and normalization preprocessing. According to the universal unmanned aerial vehicle end-to-end intelligent navigation method based on the video multi-mode large model, the environment is directly perceived through the video multi-mode large model, a flight control instruction is output, traditional module cascading is not needed, the system complexity and delay are remarkably reduced, the multi-mode large model fuses vision and semantics, and the navigation efficiency is improved. The method has zero-sample or small-sample adaptability to unknown scenes, and breaks through the limitation that a traditional method depends on a high-precision map or a pre-training model.

Description

Technical Field

[0001] This invention relates to the field of intelligent navigation technology for unmanned aerial vehicles (UAVs), specifically to a general end-to-end intelligent navigation method for UAVs based on a video multimodal large model. Background Technology

[0002] Autonomous navigation of unmanned aerial vehicles (UAVs) is one of the core challenges in the intersection of robotics and artificial intelligence. The goal is to enable UAVs to understand their environment, plan their paths, and safely reach their target locations without real-time human intervention. Traditional UAV navigation methods typically rely on a layered or modular architecture, which can be roughly divided into independent modules such as perception, localization, mapping, path planning, and control. These methods heavily depend on pre-built high-precision maps, accurate GPS signals, and robust scene recognition algorithms. With the rise of embodied intelligence research, visual-language navigation has received widespread attention as a core embodied navigation task. This task requires intelligent agents to understand natural language instructions, perceive visual information in the environment, and plan paths to reach their target locations.

[0003] Existing visual language navigation technologies can be broadly categorized into end-to-end navigation technologies and waypoint prediction-based navigation technologies. Waypoint-based methods employ a modular design, typically including modules for waypoint prediction, sub-objective planning, and motion control, offering high reliability and performance. However, their complex module design makes direct deployment on robotic platforms impractical, and waypoint prediction often relies on local observations, lacking a global perspective for long-term planning. In contrast, end-to-end methods directly predict underlying control actions from perceptual inputs and optimize them through imitation learning or reinforcement learning. This architecture simplifies deployment, and its powerful learning capabilities hold the potential to bridge the reality gap. However, these methods are limited by the scale of model parameters and the scarcity of training data. Learning the complex skills required to effectively bridge the reality gap demands extremely high computational resources, and their generalization capabilities face significant challenges.

[0004] Most existing studies focus on individual intelligent navigation indoors and outdoors, with few combining indoor and outdoor navigation methods. Furthermore, these methods all face several core common challenges: (1) Due to the limitations of model size and data scarcity, professional models have limited generalization ability and urgently need to be based on models that integrate massive amounts of real-world knowledge. (2) How to effectively maintain the key visual historical information required for decision-making under limited computing resources is a key bottleneck affecting the actual implementation of the technology; (3) There is a lack of a universal navigation agent that can handle both fine indoor operations and long-distance outdoor navigation. Summary of the Invention

[0005] The purpose of this invention is to provide a general-purpose end-to-end intelligent navigation method for unmanned aerial vehicles (UAVs) based on a video multimodal large model. By deeply integrating indoor and outdoor navigation capabilities, a unified general-purpose UAV navigation agent with rich real-world knowledge support is constructed, capable of performing precise indoor operations and long-distance outdoor navigation. The video multimodal large model directly perceives the environment and outputs flight control commands, eliminating the need for traditional module cascading, significantly reducing system complexity and latency. The multimodal large model integrates vision and semantics, possessing zero-sample or small-sample adaptability to unknown scenarios, overcoming the limitations of traditional methods that rely on high-precision maps or pre-trained models.

[0006] To achieve the above effects, the present invention provides the following technical solution: a general-purpose end-to-end intelligent navigation method for unmanned aerial vehicles based on a video multimodal large model, comprising the following steps: Step 1: Text Processing: The input text is processed by a text encoder to convert it into text words.

[0007] Step 2: Multimodal data acquisition: Obtain visual observation information from the UAV.

[0008] Step 3: Visual observation information extraction: Extract key frames of the UAV's visual observation information through a video encoder, and perform noise reduction, image stabilization, and normalization preprocessing to generate a standardized multimodal video frame sequence.

[0009] Step 4: Multimodal Feature Fusion: The standardized multimodal video frame sequence and the user-issued task instructions are fused and input into the multimodal large model. The fusion is performed through a cross-modal attention mechanism to generate a unified environmental representation vector.

[0010] Step 5: End-to-end navigation decision generation: After embedding the environmental representation vector into the text template, the multimodal large model outputs the UAV flight action control commands.

[0011] Furthermore, in step one, the text information processed by the text encoder is represented as follows: in, This indicates system prompts and task commands. d represents the total number of text terms, and d represents the dimension of the large model.

[0012] Furthermore, in step two, the UAV visual observation information includes historical image information. and current image information The UAV's visual observation information includes RGB images, thermal images, and depth images, and the airborne sensors include a visible light camera, an infrared thermal imager, and a depth camera.

[0013] Furthermore, in step three, the historical image information Divided into according to small boxes Memory Box Each memory cell i consists of the same number of adjacent observed images, the current image information. The image is divided into several patches based on its resolution. , recorded as It retains the scene information at the current moment.

[0014] Furthermore, in step four, the extraction of visual observation information includes: S301. Historical image information extraction: Historical image information is processed by the video encoder. Each memory box i obtains visual embedding. This ensures that the adjacent scene information in each memory box i is fully spatiotemporally modeled. Visual lexical units of memory box i are obtained by projecting them into a space consistent with the instruction lexical units through lexical merging and cross-modal projector. The visual lexical units of memory box i are then concatenated to obtain the historical visual lexical units. ; S302, Current Image Information Extraction: Processing the current image information through the video encoder. Each memory box i obtains visual embedding. This ensures that the adjacent scene information in each memory box i is fully spatiotemporally modeled. Through lexical merging and MLP projection, the visual lexical units are projected into a space consistent with the instruction lexical units. After the concat operation, the current visual lexical units are compressed and adjusted using a grid pool to obtain the current visual lexical units. .

[0015] Furthermore, in step S301, the historical visual lexical units are obtained. The process is as follows: Where V represents the video encoder, and P consists of a token merging operation and an MLP layer.

[0016] Furthermore, in step S302, the current visual lexical unit is obtained. The process is as follows: Gridpool uses global average pooling.

[0017] Furthermore, in step five, the environmental representation vector embedded in the text template is represented as follows: in, This indicates system prompts and task commands. Depend on It consists of several image frames, representing historical visual observations. This represents the image information received by the agent at the current moment.

[0018] Furthermore, in step five, based on the historical visual lexical units... Current visual lexical units as well as The concatenation yields a new input vector: The final model predicts actions based on vector H using the Qwen2-7B pedestal model, and outputs the actions at the next k time steps. .

[0019] Furthermore, in step five, the set of actions for the UAV flight action control commands in indoor scenarios is stop, forward, turn left and turn right, and the set of actions for the UAV flight action control commands in outdoor scenarios is forward, turn left and turn right, ascend, descend, stop, move left and move right.

[0020] This invention provides a general end-to-end intelligent navigation method for unmanned aerial vehicles (UAVs) based on a large video multimodal model, which has the following advantages: (1) The environment can be directly perceived and flight control commands can be output through video multimodal large model. There is no need for traditional module cascading, which significantly reduces system complexity and latency. The multimodal large model integrates vision and semantics and has the ability to adapt to unknown scenarios with zero or few samples, breaking through the limitations of traditional methods that rely on high-precision maps or pre-trained models.

[0021] (2) By designing switchable text prompt word templates and defining indoor and outdoor action sets respectively, the navigation framework of the multimodal large model in two significantly different environments, indoor and outdoor, is unified, enabling a single model to understand and execute instructions in different scenarios without retraining or deploying different models for specific environments, which significantly enhances the versatility and deployment flexibility of the system.

[0022] (3) By organizing historical visual observations into memory boxes and performing spatiotemporal modeling through video encoders, dynamic and contextual information between historical frames is effectively extracted, preserving the memory of the past environment. Furthermore, by compressing projection to balance information preservation and computational burden, the model can make coherent and intelligent navigation decisions based on a longer temporal context, effectively addressing the problem of long-term dependency.

[0023] (4) A video encoder is used to process visual information and a cross-modal attention mechanism is used for time synchronization and spatial calibration. This effectively integrates visual observation and airborne sensor data, eliminates time deviation and scale difference between modalities, and generates a unified and accurate environmental representation vector, providing a high-quality multimodal perception foundation for subsequent decision-making.

[0024] (5) When processing high-resolution current images, block encoding combined with Grid Pool is used for compression adjustment. While preserving key details of the current scene as much as possible, the number of input visual words is effectively controlled, avoiding the problem of historical information being overwhelmed due to overload of current frame information or excessive model computation overhead, ensuring that the UAV can perform real-time reasoning and response in complex environments.

[0025] (6) The multimodal perception is directly mapped to the flight action control command, forming a complete end-to-end navigation decision generation closed loop. This avoids the error accumulation and complex parameter tuning problems caused by the series connection of multiple independent modules such as perception, mapping, path planning and control in traditional methods, simplifies the system architecture and improves the intelligence and coordination of the overall decision.

[0026] (7) Using the Qwen2-7B large-scale pre-trained model as the base for action prediction, it makes full use of the powerful language understanding, logical reasoning and context learning capabilities of the large model, enabling it to accurately understand complex natural language task instructions. It also indicates that the method has good task generalization ability and can adapt to more diverse and complex navigation tasks in the future through prompt word engineering without changing the core structure of the model. Attached Figure Description

[0027] Figure 1 The flowchart below illustrates a general end-to-end intelligent navigation method for unmanned aerial vehicles (UAVs) based on a large video multimodal model, according to the present invention. Figure 2 This is a flowchart illustrating the visual observation information extraction process of a general-purpose end-to-end intelligent navigation method for unmanned aerial vehicles based on a video multimodal large model, as described in this invention. Figure 3 This is a schematic diagram of a general-purpose end-to-end intelligent navigation method for unmanned aerial vehicles based on a large video multimodal model according to the present invention; Figure 4 This is a schematic diagram of a task instruction template for indoor and outdoor navigation in a general-purpose UAV end-to-end intelligent navigation method based on a video multimodal large model according to the present invention. Detailed Implementation

[0028] Please see Figure 1-4 This invention provides a technical solution: a general-purpose end-to-end intelligent navigation method for unmanned aerial vehicles based on a video multimodal large model, comprising the following steps: Step 1: Text Processing: The input text is processed by a text encoder to convert it into text words.

[0029] Step 2: Multimodal data acquisition: Obtain visual observation information from the UAV.

[0030] Step 3: Visual observation information extraction: Extract key frames of UAV visual observation information through video encoder, and perform noise reduction, image stabilization and normalization preprocessing to generate a standardized multimodal video frame sequence.

[0031] Step 4: Multimodal Feature Fusion: The standardized multimodal video frame sequence and the user-issued task instructions are fused and input into the multimodal large model. The fusion is performed through a cross-modal attention mechanism to generate a unified environmental representation vector.

[0032] Step 5: End-to-end navigation decision generation: After embedding the environmental representation vector into the text template, the multimodal large model outputs the UAV flight action control commands.

[0033] Specifically, in step one, the text information processed by the text encoder is represented as follows: in, This indicates system prompts and task commands. d represents the total number of text terms, and d represents the dimension of the large model.

[0034] Specifically, in step two, the UAV visual observation information includes historical image information. and current image information The visual observation information of the UAV includes RGB images, thermal images and depth images, and the airborne sensors include visible light cameras, infrared thermal imagers and depth cameras.

[0035] Specifically, in step three, historical image information Divided into according to small boxes Memory Box Each memory box i consists of the same number of adjacent observed images, and the current image information. The image is divided into several patches based on its resolution. , recorded as It retains the scene information at the current moment.

[0036] Specifically, in step four, the extraction of visual observation information includes: S301. Historical Image Information Extraction: Processing historical image information through a video encoder. Each memory box i obtains visual embedding. This ensures that the adjacent scene information in each memory box i is fully spatiotemporally modeled. Visual lexical units of memory box i are obtained by projecting them into a space consistent with the instruction lexical units through lexical merging and cross-modal projector. The visual lexical units of memory box i are then concatenated to obtain the historical visual lexical units. ; S302. Current Image Information Extraction: Processing current image information through a video encoder. Each memory box i obtains visual embedding. This ensures that the adjacent scene information in each memory box i is fully spatiotemporally modeled. Through lexical merging and MLP projection, the visual lexical units are projected into a space consistent with the instruction lexical units. After the concat operation, the current visual lexical units are compressed and adjusted using a grid pool to obtain the current visual lexical units. .

[0037] Specifically, in step S301, historical visual lexical units are obtained. The process is as follows: Where V represents the video encoder, and P consists of a token merging operation and an MLP layer.

[0038] Specifically, in step S302, the current visual lexical unit is obtained. The process is as follows: Gridpool uses global average pooling.

[0039] Specifically, in step five, the environmental representation vector embedded in the text template is represented as follows: in, This indicates system prompts and task commands. Depend on It consists of several image frames, representing historical visual observations. This represents the image information received by the agent at the current moment.

[0040] Specifically, in step five, based on historical visual lexical units... Current visual lexical units as well as The concatenation yields a new input vector: The final model predicts actions based on vector H using the Qwen2-7B pedestal model, and outputs the actions at the next k time steps. .

[0041] Specifically, in step five, the set of drone flight action control commands in indoor scenarios is stop, forward, turn left and turn right, while the set of drone flight action control commands in outdoor scenarios is forward, turn left and turn right, ascend, descend, stop, move left and move right.

[0042] This invention provides a general end-to-end intelligent navigation method for unmanned aerial vehicles (UAVs) based on a video multimodal large model.

[0043] The multimodal large model mainly consists of three parts: a text processor, a vision processor, and a large model base.

[0044] Unlike existing multimodal large-scale model research that uses separate text and image input structures, this model adopts a multimodal input structure, embedding visual information from UAV observations into a text template, which can be represented in the following form: (1) in, This indicates system prompts and task commands. Depend on It consists of several image frames, representing historical visual observations. This represents the image information received by the agent at the current moment. This multimodal cue word template helps large models better understand the context and task.

[0045] Belongs to X, which consists of three parts: text Current image term and historical image terms Their composition relationship is as follows: Figure 3 (History frame) represents a history word. (Current frame) indicates The remaining text indicates The subscript indicates which word it is.

[0046] Based on the above input information, 1. The text processor processes the text portion of the input, converting it into text words, which are then used by the base model to read instructions.

[0047] Because the indoor and outdoor environments are combined, and their action sets are different, two navigation templates are designed to distinguish between indoor and outdoor scenes. Specifically: (1) For indoor environments, the text template is switched to: "You are an embodied navigation agent deployed in an indoor environment. Your task is: ... Based on the task description, predict the next k actions to complete the task using previously observed video frames (historical frames) and the current visual input (current frame). You must respond using only numbers from the following action set: (0: stop, 1: forward, 2: turn left, 3: turn right)"; (2) For outdoor environments, the text template is switched to: "You are an embodied navigation agent deployed in an indoor / outdoor environment. Your task is: ... Based on the task description, predict the next k actions to complete the task using previously observed video frames (historical frames) and the current visual input (current frame). You must respond using only numbers from the following action set: (0: stop, 1: forward, 2: turn left, 3: turn right, 4: up, 5: down, 6: move left, 7: move right)".

[0048] Regarding the text processing section, the text information processed by the text encoder can be obtained. , This indicates system prompts and task commands, here d is the total number of text terms, and d is the dimension of the large model.

[0049] 2. Visual Observation Information Extraction. Successful indoor navigation work inspires us to combine historical information and current observations to help agents better complete long-term navigation tasks by classifying visual observations into current and historical observations. This also takes into account the similarity of the agent's environment at adjacent moments.

[0050] 2.1 Therefore, historical image information can be... Divided into according to small boxes Memory Box Each memory cell consists of the same number of adjacent observed images. Current image information. The image is divided into several patches based on its resolution. , recorded as This is to preserve as much scene information as possible at the current moment.

[0051] For historical observation information, a pre-trained video encoder is used to process each memory box i to obtain visual embeddings. This allows the adjacent scene information within each box to be fully spatiotemporally modeled and reflected in each historical visual token. Subsequently, By combining lexical units and projecting them onto a space consistent with the instruction lexical units through a cross-modal projector, a series of processed memory box visual lexical units are obtained. Finally, these visual lexical units are concatenated to obtain the final historical visual lexical units. This process can be expressed by the following formula: (2) (3) Here, V represents the video encoder, and P consists of a token merging operation and an MLP layer. To fully preserve historical visual information while ensuring inference and training efficiency, the visual information in each box is compressed to 64 tokens, averaging 16 tokens per historical moment, which are then combined to form... The number of historical visual tokens.

[0052] 2.2. Regarding the processing of the current observation, for multiple patch information, the processing is similar to that of historical observations, sequentially passing through a video encoder, word merging, and MLP, followed by a concat operation. However, due to the relatively large number of current visual words obtained, approximately... To balance the size of historical visual words and current visual words, a grid pool is used to compress the current visual words.

[0053] The visual word compression can be adjusted by setting the grid size. The entire operation process is shown below: (4) (5) Gridpool uses global average pooling.

[0054] 3. Based on 2.1 above... And obtained from 2.2 , and 1 obtained The new input vector is obtained by concatenating the corresponding positions in equation (1): (6) The final model predicts actions based on vector H using the Qwen2-7B pedestal model, and outputs the actions at the next k time steps. .

[0055] Qwen2-7B is a large language model based on the Transformer architecture. It uses grouped query attention and bi-block attention techniques to optimize long text processing capabilities. The model was pre-trained on a dataset of 7 trillion tokens covering 30 languages, with particular emphasis on data quality in the code and mathematical domains. After training through two stages of supervised fine-tuning and direct preference optimization, the model can be adapted to tasks such as text generation and code interpretation. It supports cross-modal application development through fine-tuning techniques such as LoRA and has become one of the most active medium-sized open-source language models in the development ecosystem.

[0056] Given the inherent differences in action space between indoor and outdoor scenes, in order to ensure high consistency with the original characteristics of each scene and facilitate accurate evaluation of cross-scene performance, this invention specifically designs two independent action sets that are consistent with their respective environments: in indoor scenes, the action set used is [stop, forward, turn left, turn right], while in outdoor scenes, the action set is expanded to [forward, turn left, turn right, rise, fall, stop, move left, move right].

[0057] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A general-purpose end-to-end intelligent navigation method for unmanned aerial vehicles (UAVs) based on a video multimodal large model, characterized in that, Includes the following steps: S1. Text Processing: The input text is processed by a text encoder and converted into text words. S2. Multimodal data acquisition: Acquiring visual observation information from the UAV; S3. Visual observation information extraction: The key frames of the UAV's visual observation information are extracted by the video encoder and preprocessed by denoising, stabilizing, and normalizing to generate a standardized multimodal video frame sequence. S4. Multimodal feature fusion: The standardized multimodal video frame sequence and the user-issued task instructions are fused and input into the multimodal large model. The fusion is performed through a cross-modal attention mechanism to generate a unified environmental representation vector. S5. End-to-end navigation decision generation: After embedding the environmental representation vector into the text template, the multimodal large model outputs the UAV flight action control commands.

2. The general end-to-end intelligent navigation method for unmanned aerial vehicles based on a video multimodal large model according to claim 1, characterized in that, In step S1, the text information processed by the text encoder is represented as follows: in, This indicates system prompts and task commands. d represents the total number of text terms, and d represents the dimension of the large model.

3. The general end-to-end intelligent navigation method for unmanned aerial vehicles based on a video multimodal large model according to claim 1, characterized in that, In step S2, the UAV visual observation information includes historical image information. and current image information The UAV's visual observation information includes RGB images, thermal images, and depth images, and the airborne sensors include a visible light camera, an infrared thermal imager, and a depth camera.

4. The general end-to-end intelligent navigation method for unmanned aerial vehicles based on a video multimodal large model according to claim 1, characterized in that, In step S3, the historical image information Divided into according to small boxes Memory Box Each memory cell i consists of the same number of adjacent observed images, the current image information. The image is divided into several patches based on its resolution. , recorded as It retains the scene information at the current moment.

5. The general end-to-end intelligent navigation method for unmanned aerial vehicles based on a video multimodal large model according to claim 1, characterized in that, In step S3, the extraction of visual observation information includes: S301. Historical image information extraction: Historical image information is processed by the video encoder. Each memory box i obtains visual embedding. This ensures that the adjacent scene information in each memory box i is fully spatiotemporally modeled. Visual lexical units of memory box i are obtained by projecting them into a space consistent with the instruction lexical units through lexical merging and cross-modal projector. The visual lexical units of memory box i are then concatenated to obtain the historical visual lexical units. ; S302, Current Image Information Extraction: Processing the current image information through the video encoder. Each memory box i obtains visual embedding. This ensures that the adjacent scene information in each memory box i is fully spatiotemporally modeled. Through lexical merging and MLP projection, the visual lexical units are projected into a space consistent with the instruction lexical units. After the concat operation, the current visual lexical units are compressed and adjusted using a grid pool to obtain the current visual lexical units. .

6. The general end-to-end intelligent navigation method for unmanned aerial vehicles based on a video multimodal large model according to claim 1, characterized in that, In step S301, the historical visual lexical units are obtained. The process is as follows: Where V represents the video encoder, and P consists of a token merging operation and an MLP layer.

7. A general end-to-end intelligent navigation method for unmanned aerial vehicles based on a video multimodal large model according to claim 1, characterized in that, In step S302, the current visual lexical unit is obtained. The process is as follows: Gridpool uses global average pooling.

8. A general end-to-end intelligent navigation method for unmanned aerial vehicles based on a video multimodal large model according to claim 1, characterized in that, In step S5, the environmental representation vector embedded in the text template is represented as follows: in, This indicates system prompts and task commands. Depend on It consists of several image frames, representing historical visual observations. This represents the image information received by the agent at the current moment.

9. A general end-to-end intelligent navigation method for unmanned aerial vehicles based on a video multimodal large model according to claim 1, characterized in that, In step S5, based on the historical visual lexical units Current visual lexical units as well as The concatenation yields a new input vector: The final model predicts actions based on vector H using the Qwen2-7B pedestal model, and outputs the actions at the next k time steps. .

10. A general-purpose end-to-end intelligent navigation method for unmanned aerial vehicles based on a video multimodal large model according to claim 1, characterized in that, In step S5, the set of actions for the UAV flight action control command in an indoor scene is stop, forward, turn left and turn right, and the set of actions for the UAV flight action control command in an outdoor scene is forward, turn left and turn right, ascend, descend, stop, move left and move right.

Images