Unmanned aerial vehicle intelligent body training method based on multi-modal large language model

Abstract

The invention discloses an unmanned aerial vehicle intelligent agent training method based on a multi-modal large language model, and relates to the technical field of unmanned aerial vehicle intelligent agents, and the method comprises the steps: image preprocessing: zooming an original image collected by an unmanned aerial vehicle to a fixed size, and carrying out the standardization, according to the invention, an image preprocessing module, a multi-modal processing module and a joint training module are arranged, an original image collected by an unmanned aerial vehicle is zoomed to a fixed size and standardized, image block features are extracted through a visual encoder, and through triple technical innovations of Q-Former cross-modal alignment, dynamic loss weighting and LoRA lightweight, the real-time performance of the unmanned aerial vehicle is improved, and the real-time performance of the unmanned aerial vehicle is improved. The problems of poor environmental adaptability, difficulty in end-side deployment and multi-task conflict in unmanned aerial vehicle agent training are solved, unmanned aerial vehicle agent training data and corresponding analysis results are managed, visualized and stored, unmanned aerial vehicle agent training source management can be realized through internet-of-things cloud management and control, and the training efficiency is improved. And the intelligent level of unmanned aerial vehicle intelligent body training management is improved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) intelligent agent technology, specifically to a UAV intelligent agent training method based on a multimodal large language model. Background Technology

[0002] In recent years, unmanned aerial vehicles (UAVs), as a flexible and efficient aerial platform, have been widely used in various fields such as security patrol, emergency rescue, intelligent logistics, agricultural plant protection, and geographic surveying. With the increasing complexity of tasks, traditional UAV control methods relying on rule-based programming or shallow learning algorithms are no longer sufficient to meet the demands for autonomous decision-making and intelligent task execution in complex environments. Therefore, UAV intelligent agents integrating advanced artificial intelligence technologies have become a research hotspot.

[0003] Driven by deep learning and reinforcement learning, significant progress has been made in intelligent control methods for unmanned aerial vehicles (UAVs) based on visual perception and policy optimization. However, most existing methods still rely on information input from a single modality (such as images or radar), making it difficult to achieve semantic understanding, multi-source information fusion, and human-machine collaborative interaction in complex scenarios. This results in insufficient robustness and generalization ability of the system in changing environments.

[0004] In recent years, large language models (LMs) such as GPT-4, Gemini, and Claude have demonstrated outstanding performance in natural language processing, reasoning, and knowledge representation. In particular, multimodal large language models (LMs) can process information from multiple modalities such as images, text, and speech simultaneously, and have powerful semantic understanding, reasoning, and generation capabilities, providing new solutions for ubiquitous perception and complex decision-making of UAV intelligent agents.

[0005] In existing technologies, UAV agent training suffers from poor environmental adaptability, difficulty in edge deployment, and multi-task conflicts. It cannot effectively integrate multi-source information and perform deep semantic modeling, which reduces the intelligence level and task adaptability of UAVs in practical applications, makes it inconvenient for staff to conduct remote supervision, and causes many inconveniences. Therefore, this invention needs to design a UAV agent training method based on a multimodal large language model to solve the above-mentioned problems. Summary of the Invention The purpose of this invention is to provide a training method for UAV intelligent agents based on a multimodal large language model in order to solve the above-mentioned problems, thereby solving the problems mentioned in the background art.

[0006] To address the above problems, the present invention provides a technical solution: A method for training UAV agents based on a multimodal large language model includes the following specific steps: S1. Image preprocessing: Scaling the raw images captured by the drone to a fixed size and standardizing them; S2, Multimodal coding: A visual encoder extracts image patch features; The Q-Former module compresses visual features into fixed-length tokens; Text encoders process natural language instructions; Vision encoder stage (ViT): Input: The processed image Output: PatchTokens Adjustable parameters: model size, PatchSize; S3, Cross-modal fusion: Concatenates visual tokens and text tokens, embedding modal identifiers and location codes; S4. Multimodal LLM processing: Autoregressive modeling is performed using the Transformer backbone; S5. Joint Training: Calculate multi-task loss based on visual localization, navigation, and question answering tasks, and update model parameters through backpropagation.

[0007] In a preferred embodiment of the present invention, the image preprocessing in step S1 specifically includes the following steps: S101, Data Input Stage: Input a JPEG image in its original format; S102, Preprocessing steps: Resize: Scales the device to a fixed size. Normalize: Standardizes the RGB channels by mean / variance; Optional enhancements: such as RandomCrop, Flip; S103, Text Input; Tokenization: For example, using the same tokenizer as Qwen2.5; Cut off and fill to the maximum length.

[0008] In a preferred embodiment of the present invention, the Q-Former module in step S2 is used to input the visual tokens output by ViT, and uses a learnable Querytoken to perform cross-attention with the ViT output; The Q-Former module in step S2 is also used to output a fixed number of visual token representations, compressing variable-length image representations into fixed-structure visual cues for use by the language model. The parameters are adjustable: number of queries and Q-Former depth. The text encoder in step S2 is used to input the tokenID sequence of the text Prompt after it has been processed by the Tokenizer, and is also used to process the lookup table to obtain the TokenEmbedding.

[0009] In a preferred embodiment of the present invention, the multimodal LLM backbone in step S4 is based on a Transformer decoder-only structure and is used to input the fused token sequence. The multimodal LLM backbone in step S4 is also used to process the token sequence after multi-layer Transformer processing and fusion, and to use CausalMask to achieve autoregressive modeling. The parameters are adjustable: number of layers, hidden dimension, number of attention heads, FFN dimension, RMSnorm and other details.

[0010] In a preferred embodiment of the present invention, step S5 includes the following specific steps: S501, Training Objective and Loss Function (1) Training task: Follow the text and image instructions; Image description generation, visual localization; Image and text Q&A; (2) Loss types: CrossEntropyLoss, weighted combinatorial multi-task Loss; (3) Adjustable parameters: Loss weight ratio, LabelSmoothing and other regularization terms; S502, Backpropagation and Optimizer Update (1) The loss is backpropagated to all trainable parameters; (2) Optimizer: Commonly used: AdamW, 8-bit Adam, supports weight decay; (3) Learning rate schedulers: Cosine, LinearWarmup, Exponential, etc.; (4) Parameter update strategy: full parameter fine-tuning, plug-in LoRA fine-tuning; (5) Adjustable parameters: learning rate, LoRARank, BatchSize, GradientAccumulationSteps.

[0011] In a preferred embodiment of the present invention, step S3 includes the following specific steps: S301. Concatenate the visual token and text token output by Q-Former; S302, Add special ModalityEmbedding and position encoding; S303. The concatenated sequence is fed into the backbone of the language model as the overall input.

[0012] In a preferred embodiment of the present invention, before performing the image preprocessing in step S1, it is necessary to construct a UAV intelligent agent training system. The UAV intelligent agent training system includes an image preprocessing module, a multimodal coding module, a cross-modal fusion module, a multimodal processing module, and a joint training module.

[0013] In a preferred embodiment of the present invention, the output of the image preprocessing module is communicatively connected to the input of the multimodal coding module, the output of the multimodal coding module is communicatively connected to the input of the cross-modal fusion module, the output of the cross-modal fusion module is communicatively connected to the input of the multimodal processing module, and the output of the multimodal processing module is communicatively connected to the input of the joint training module.

[0014] In a preferred embodiment of the present invention, the image preprocessing module is used to scale the original image captured by the UAV to a fixed size and standardize it for processing the original image data captured by the UAV camera, such as denoising, enhancement, size normalization and format conversion, so as to improve the quality of image input and adapt to the input requirements of multimodal models. The multimodal coding module is used to extract features from multimodal inputs such as images, text, speech, and sensor data. Image modality extracts semantic features through a visual encoder, text modality extracts semantic vectors through a pre-trained language model, speech modality is converted into text through a speech recognition model and then encoded, and sensor data is encoded into feature vectors after time-series modeling. The cross-modal fusion module is used to align and semantically associate the encoding results of each modality based on attention mechanisms or transformer structures, thereby achieving deep fusion and information complementarity between different modalities and constructing a unified representation vector.

[0015] In a preferred embodiment of the present invention, the multimodal processing module is used to perform dimensionality reduction, normalization and semantic enhancement processing on the fused feature vector to ensure the robustness and interpretability of the features and further improve the ability of the large language model to understand the fused information. The joint training module is used to perform task-driven end-to-end training of the UAV agent based on a joint optimization strategy of reinforcement learning and supervised learning. It achieves adaptive learning of intelligent behaviors such as path planning, target recognition, and dynamic decision-making through interaction with a multimodal large language model.

[0016] The beneficial effects of this invention are as follows: By setting up an image preprocessing module, a multimodal coding module, a cross-modal fusion module, a multimodal processing module, and a joint training module, this invention constructs a complete UAV intelligent agent training system. During actual operation, the original images acquired by the UAV are scaled to a fixed size and standardized. Image patch features are extracted through a visual encoder, the Q-Former module compresses visual features into fixed-length tokens, a text encoder processes natural language instructions, concatenates visual tokens and text tokens, embeds modality identifiers and positional codes, and performs autoregressive modeling through a Transformer backbone based on visual localization. This invention utilizes three innovative technologies—Q-Former cross-modal alignment, dynamic loss weighting, and LoRA lightweighting—to overcome the challenges of poor environmental adaptability, difficult edge deployment, and multi-task conflicts in UAV agent training. These technologies include navigation and question-answering task calculations, and backpropagation to update model parameters. Real-world testing shows that key indicators in logistics, inspection, and disaster relief scenarios are improved by an average of over 40%, demonstrating significant technological barriers and market competitiveness. Furthermore, the invention enables the management, visualization, and storage of UAV agent training data and corresponding analysis results, facilitating the management of UAV agent training sources through IoT cloud control and improving the intelligence level of UAV agent training management. Attached Figure Description For ease of explanation, the present invention will be described in detail below with reference to specific embodiments and accompanying drawings.

[0017] Figure 1 This is an overall flowchart of a training method for UAV intelligent agents based on a multimodal large language model according to the present invention; Figure 2 This is a flowchart of the UAV agent training method based on a multimodal large language model according to the present invention. Figure 3 This is a diagram illustrating the adjustable parameters and their impact on performance of a UAV intelligent agent training method based on a multimodal large language model according to the present invention. Figure 4 This is an overview diagram of the adjustable parameters and their impact on performance of a UAV intelligent agent training method based on a multimodal large language model according to the present invention. Detailed Implementation like Figure 1 , Figure 2 , Figure 3 and Figure 4 As shown, the specific implementation adopts the following technical solution: A method for training UAV agents based on a multimodal large language model includes the following specific steps: S1. Image preprocessing: Scaling the raw images captured by the drone to a fixed size and standardizing them; S2, Multimodal coding: Visual encoders (ViT) extract image patch features; The Q-Former module compresses visual features into fixed-length tokens; Text encoders process natural language instructions; Vision Transformer (ViT) stage: Input: The processed image Output: PatchTokens (e.g., 14 × 14 = 196 tokens) Adjustable parameters: Model size (Base / Large / Huge), PatchSize (16×16 or 14×14); S3, Cross-modal fusion: Concatenates visual tokens and text tokens, embedding modal identifiers and location codes; S4. Multimodal LLM processing: Autoregressive modeling is performed using the Transformer backbone; S5. Joint Training: Calculate multi-task loss based on visual localization, navigation, and question answering tasks, and update model parameters through backpropagation.

[0018] In a preferred embodiment of the present invention, the image preprocessing in step S1 specifically includes the following steps: S101, Data Input Stage: Input a JPEG image in its original format; S102, Preprocessing steps: Resize: Scale uniformly to a fixed size (512×512); Normalize: Standardizes the RGB channels by mean / variance; Optional enhancements: such as RandomCrop, Flip (used only during training). S103, Text Input: Format: Natural language Prompt, for example, " \n[grounding]Agreytruckwasparkedonthesideoftheroad,ayellowtruckatthetopleft."; Tokenization: For example, using the same tokenizer as Qwen2.5 (based on BPE); Truncate and pad to the maximum length (e.g., 32768 token).

[0019] In a preferred embodiment of the present invention, the Q-Former module in step S2 is used to input the visual tokens output by ViT, and uses the learnable Querytoken to perform cross-attention with the ViT output. The Q-Former module in step S2 is also used to output a fixed number of visual token representations (e.g., 32), compressing the variable-length image representation into a fixed-structure visual cue for use by the language model. The parameters are adjustable: number of queries (e.g., 16 / 32 / 64) and Q-Former depth (number of Transformer layers). The text encoder in step S2 is used to input the tokenID sequence of the text Prompt after it has been processed by the Tokenizer, and is also used to process the lookup table to obtain the TokenEmbedding (position encoding, type encoding, etc.).

[0020] In a preferred embodiment of the present invention, the multimodal LLM backbone (Transformer) in step S4 is based on a Transformer decoder-only structure (similar to ChatGPT) and used as the input fused token sequence. The multimodal LLM backbone in step S4 is also used to process the token sequence after multi-layer Transformer processing and fusion, and to implement autoregressive modeling using CausalMask. The parameters are adjustable: number of layers (e.g., 32 / 40 / 60), hidden dimension (e.g., 4096 / 8192), number of attention heads (e.g., 32 / 64), FFN dimension, RMSNorm and other details.

[0021] In a preferred embodiment of the present invention, step S5 includes the following specific steps: S501, Training Objective and Loss Function (1) Training task: Instruction following (used for visual language navigation); Image caption generation, visual positioning (Captioning, VisualGrounding). Visual Q&A (VQA) (2) Loss types: CrossEntropyLoss (main task), weighted combination multi-task Loss (VisualGroundingloss + VQAloss + Navigationloss). (3) Adjustable parameters: Loss weight ratio (e.g., VisualGrounding: 1.0, VQA: 0.5, Navigation: 1.0), LabelSmoothing and other regularization terms; S502, Backpropagation and Optimizer Update (1) The loss is backpropagated to all trainable parameters (ViT / Q-Former / LLM or LoRA fine-tuning). (2) Optimizer: Commonly used: AdamW, 8-bit Adam, supports weight decay; (3) Learning rate scheduler: Cosine, LinearWarmup, Exponential, etc.; (4) Parameter update strategy: Full-Fine-tuning, plug-in LoRA fine-tuning (training only the inserted LoRA layer); (5) Adjustable parameters: learning rate (e.g., 5e-5), LoRARank (e.g., 4 / 8 / 16), BatchSize (e.g., 64 / 128), GradientAccumulationSteps.

[0022] In a preferred embodiment of the present invention, step S3 includes the following specific steps: S301. Concatenate the visual token and text token output by Q-Former (Concat); S302, Add special ModalityEmbedding (visual / text identifier) ​​and location encoding; S303. The concatenated sequence is fed into the backbone of the language model as the overall input.

[0023] In a preferred embodiment of the present invention, before performing image preprocessing in step S1, a UAV intelligent agent training system needs to be constructed. The UAV intelligent agent training system includes an image preprocessing module, a multimodal coding module, a cross-modal fusion module, a multimodal processing module, and a joint training module. The output of the image preprocessing module is communicatively connected to the input of the multimodal coding module, the output of the multimodal coding module is communicatively connected to the input of the cross-modal fusion module, the output of the cross-modal fusion module is communicatively connected to the input of the multimodal processing module, and the output of the multimodal processing module is communicatively connected to the input of the joint training module.

[0024] In a preferred embodiment of the present invention, the image preprocessing module is used to scale the original images acquired by the UAV to a fixed size and standardize them. This is used to perform denoising, enhancement, size normalization, and format conversion on the original image data acquired from the UAV camera, thereby improving the quality of the image input and adapting it to the input requirements of the multimodal model. The multimodal encoding module is used to extract features from multimodal inputs such as images, text, speech, and sensor data. Image modality uses a visual encoder to extract semantic features; text modality uses a pre-trained language model to extract semantic vectors; speech modality uses a speech recognition model to convert it into text and then encodes it; and sensor data is encoded into features after time-series modeling. The cross-modal fusion module is used to align and semantically model the encoding results of each modality based on attention mechanisms or transformer structures, to achieve deep fusion and information complementarity between different modalities, and to construct a unified representation vector. The multimodal processing module is used to perform dimensionality reduction, normalization and semantic enhancement processing on the fused feature vector to ensure the robustness and interpretability of the features, and further improve the understanding ability of the large language model of fused information. The joint training module is used to perform task-driven end-to-end training of the UAV agent based on a joint optimization strategy of reinforcement learning and supervised learning, and to achieve adaptive learning of intelligent behaviors such as path planning, target recognition and dynamic decision-making through interaction with the multimodal large language model.

[0025] Example When the energy cloud management system is connected to the factory management system: S1. After the staff conducts a unified inspection of the on-site and remote power equipment and confirms that they can all start normally, the UAV intelligent agent training system is started. Image preprocessing: The original images collected by the UAV are scaled to a fixed size and standardized. The UAV intelligent agent training system controls the image preprocessing module to scale the original images collected by the UAV to a fixed size and standardize them. This module is used to perform denoising, enhancement, size normalization and format conversion on the original image data collected by the UAV camera to improve the quality of image input and adapt to the input requirements of the multimodal model. S2. Multimodal Encoding: The visual encoder extracts image patch features; the Q-Former module compresses visual features into fixed-length tokens; the text encoder processes natural language commands; the UAV intelligent agent training system controls the multimodal encoding module to extract features from multimodal inputs such as images, text, speech, and sensor data. Image modality extracts semantic features through the visual encoder; text modality extracts semantic vectors through a pre-trained language model; speech modality is converted into text through a speech recognition model and then encoded; sensor data is encoded into feature vectors after time-series modeling. S3. Cross-modal fusion: By splicing visual tokens and text tokens and embedding modal identifiers and position codes, the cross-modal fusion module of the UAV intelligent agent training system is controlled by an attention mechanism or transformer structure to align and semantically model the encoding results of each modality, thereby achieving deep fusion and information complementarity between different modalities and constructing a unified representation vector. S4. Multimodal LLM processing: Autoregressive modeling is performed through the Transformer backbone. The UAV intelligent agent training system controls the multimodal processing module to perform dimensionality reduction, normalization and semantic enhancement on the fused feature vectors to ensure the robustness and interpretability of the features and further improve the ability of the large language model to understand the fused information. S5. Joint Training: Based on visual positioning, navigation, and question answering tasks, multi-task loss is calculated, and backpropagation updates model parameters. The UAV agent training system controls the joint training module to perform task-driven end-to-end training on the UAV agent based on a joint optimization strategy of reinforcement learning and supervised learning. Through interaction with a multimodal large language model, adaptive learning of intelligent behaviors such as path planning, target recognition, and dynamic decision-making is achieved.

[0026] Specifically, in practical applications, multiple image preprocessing modules are used in conjunction with a multimodal coding module, a cross-modal fusion module, a multimodal processing module, and a joint training module. These image preprocessing modules are located in different geographical locations. This invention constructs a complete UAV intelligent agent training system by setting up image preprocessing, multimodal coding, cross-modal fusion, multimodal processing, and joint training modules. During actual operation, the original images captured by the UAV are scaled to a fixed size and standardized. Image patch features are extracted using a visual encoder, the Q-Former module compresses the visual features into fixed-length tokens, a text encoder processes natural language instructions, and the visual and text tokens are concatenated and embedded with modal identifiers. This invention employs location encoding and autoregressive modeling via a Transformer backbone. It calculates multi-task losses based on visual positioning, navigation, and question-answering tasks, and updates model parameters through backpropagation. Through three technological innovations—Q-Former cross-modal alignment, dynamic loss weighting, and LoRA lightweighting—this invention overcomes the problems of poor environmental adaptability, difficult edge deployment, and multi-task conflicts in UAV agent training. Real-world testing data shows that key indicators in scenarios such as logistics, inspection, and disaster relief are improved by an average of over 40%, demonstrating significant technological barriers and market competitiveness. Managing, visualizing, and storing UAV agent training data and corresponding analysis results facilitates the management of UAV agent training sources through IoT cloud control, thereby improving the intelligence level of UAV agent training management.

[0027] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0028] The modules for image preprocessing, multimodal coding, cross-modal fusion, multimodal processing, and joint training may or may not be physically separate. The components displayed as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0029] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0030] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program instructions, such as USB flash drives, portable hard drives, read-only storage servers, random access storage servers, magnetic disks, or optical disks.

[0031] Furthermore, it should be noted that the combination of the various technical features in this case is not limited to the combination methods described in the claims of this case or the combination methods described in the specific embodiments. All technical features described in this case can be freely combined or combined in any way, unless they contradict each other.

[0032] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

[0033] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training a UAV agent based on a multi-modal large language model, characterized in that, The method comprises the following specific steps: S1, image preprocessing: scaling the original image collected by the unmanned aerial vehicle to a fixed size and normalizing it; S2, multi-modal encoding: a visual encoder extracts image block features; a Q-Former module compresses the visual features into fixed-length tokens; a text encoder processes natural language instructions; S3, cross-modal fusion: splicing the visual tokens and the text tokens, embedding the modal identifier and the position encoding; S4, multi-modal LLM processing: performing autoregressive modeling through a Transformer backbone; S5, joint training: calculating a multi-task loss based on visual positioning, navigation, and question-answering tasks, and updating the model parameters through back propagation.

2. The unmanned aerial vehicle agent training method based on a multi-modal large language model according to claim 1, characterized in that: The image preprocessing in the step S1 specifically comprises the following steps: S101, data input stage: inputting a JPEG image in an original format; S102, preprocessing step: Resize: uniformly scaling to a fixed size; Normalize: normalizing the RGB channels according to the mean / variance; Optional enhancement: such as RandomCrop, Flip; S103, text input.

3. The unmanned aerial vehicle agent training method based on a multi-modal large language model according to claim 1, characterized in that: the Q-Former module in the step S2 is used to input the visual tokens output by ViT and perform cross-attention with the ViT output using a learnable Query token; the Q-Former module in the step S2 is also used to output a fixed number of visual Token representations, compressing the variable-length image representations into fixed-structure visual prompts for use by the language model; the text encoder in the step S2 is used to input the token ID sequence of the text Prompt after Tokenization and is also used to process the Token Embedding obtained through table lookup.

4. The unmanned aerial vehicle agent training method based on a multi-modal large language model according to claim 3, characterized in that: the multi-modal LLM backbone in the step S4 is based on a Transformer decoder-only structure and is used to input the fused token sequence; the multi-modal LLM backbone in the step S4 is also used to process the fused token sequence through multiple layers of Transformer processing and use a CausalMask to implement autoregressive modeling.

5. The unmanned aerial vehicle agent training method based on a multi-modal large language model according to claim 1, characterized in that: The step S5 comprises the following specific steps: S501, training target and loss function (1) training tasks: following instructions with images and texts; image description generation and visual positioning; question-answering with images and texts; (2) loss types; (3) adjustable parameters; S502, back propagation and optimizer update.

6. The method of claim 5, wherein the method further comprises: The step S3 comprises the following specific steps: S301, splicing the visual tokens output by the Q-Former with the text tokens; S302, adding special ModalityEmbedding and position encoding; S303, splicing the sequence as a whole to input into the language model backbone.

7. The unmanned aerial vehicle agent training method based on a multi-modal large language model according to claim 1, characterized in that: Before the step S1 image preprocessing is performed, it is necessary to construct a UAV intelligent agent training system, the UAV intelligent agent training system comprising an image preprocessing module, a multi-modal encoding module, a cross-modal fusion module, a multi-modal processing module and a joint training module.

8. The method of claim 7, wherein the method further comprises: The output end of the image preprocessing module is in communication connection with the input end of the multi-modal encoding module, the output end of the multi-modal encoding module is in communication connection with the input end of the cross-modal fusion module, the output end of the cross-modal fusion module is in communication connection with the input end of the multi-modal processing module, and the output end of the multi-modal processing module is in communication connection with the input end of the joint training module.

9. The multi-modal large language model-based UAV intelligent agent training method according to claim 7, characterized in that: The image preprocessing module is configured to scale and standardize the original image collected by the UAV to a fixed size, and to perform denoising, enhancement, size normalization and format conversion on the original image data collected by the UAV camera. The multi-modal encoding module is configured to extract features from multi-modal inputs such as images, texts, voices and sensor data. The cross-modal fusion module is configured to align and model the semantic association of the encoding results of each modality based on an attention mechanism or a transformer structure.

10. The multi-modal large language model-based UAV intelligent agent training method according to claim 7, characterized in that: The multi-modal processing module is configured to perform dimensionality reduction, normalization and semantic enhancement processing on the fused feature vectors. The joint training module is configured to use a joint optimization strategy based on reinforcement learning and supervised learning.

Images