A Method and System for Unmanned Aerial Vehicle (UAV) Swarm Formation Control Based on Multimodal Large Language Model

Abstract

This application provides a method and system for drone swarm formation control based on a multimodal large language model. The method includes: acquiring environmental images captured or received by a drone array; parsing the environmental images using a first large language model to generate an environmental description; acquiring user commands and environmental data; parsing the user commands and environmental data using a second large language model; and generating command semantics by combining the environmental description; acquiring a preset formation image; calling the first large language model to parse the preset formation image; generating formation commands by combining the command semantics; and controlling the drone array to complete swarm formation according to the formation commands. This application solves the problem of real-time drone swarm control based on multimodal image acquisition and recognition by using a multimodal large language model and a supervised-trained large language model.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a method and system for UAV swarm formation control based on a multimodal large language model. Background Technology

[0002] Collaborative unmanned aerial vehicle (UAV) systems are playing an increasingly important role in various fields due to their high flexibility, integrated intelligent functions, and comprehensive collaborative capabilities. By integrating high-precision sensors, efficient wireless communication technologies, and advanced autonomous decision-making algorithms, these systems can effectively execute and optimize various tasks through the coordinated cooperation of multiple UAVs in complex environments. Existing research utilizes Large Language Models (LLMs) to achieve UAV swarm control. However, existing LLM applications in UAV swarm control mostly focus on dialogue interaction. For example, invention patent application number 202311642952.9 discloses a method and system for precise command and control of UAV swarms based on LLMs. This method converts user-inputted voice / text data into a vertical LLM that can be input into the command and control domain to achieve UAV swarm control, but it cannot achieve formation control based on image formats. In the existing technology, most applications of UAV swarm control based on LLMs focus on dialogue interaction rather than multimodal image perception and understanding, and cannot achieve real-time UAV swarm control based on multimodal image acquisition and recognition.

[0003] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0004] In view of the shortcomings of the prior art, this application provides a method and system for UAV swarm formation control based on a multimodal large language model, aiming to solve the problem of the lack of UAV swarm control based on a large language model for multimodal image perception and understanding in the prior art, and to realize real-time UAV swarm control based on multimodal image acquisition and recognition.

[0005] This application is achieved through the following technical solution:

[0006] The first aspect of this application is to provide a method for controlling drone swarm formation based on a multimodal large language model, comprising the following steps:

[0007] Acquire environmental images captured or received by a drone array, and generate an environmental description by parsing the environmental images using a first language model;

[0008] The system acquires user commands and environment data, parses the user commands and environment data based on the second language model, and generates command semantics by combining the environment description.

[0009] A preset formation image is obtained, the first large language model is called to parse the preset formation image, and the formation instructions are generated by combining the command semantics. The UAV array is then controlled to complete the cluster formation according to the formation instructions.

[0010] Specifically, the drone array includes at least one lead drone and several follower drones; the lead drone is used to capture or receive the environmental images and the preset formation images; the lead drone and any of the follower drones are equipped with several sensors to acquire the environmental data.

[0011] Specifically, the first large language model is the multimodal GPT-4 large language model.

[0012] Specifically, the step of acquiring environmental images captured or received by the UAV array and generating an environmental description by parsing the environmental images using a first language model includes the following steps:

[0013] The pilot drone captures or receives the environmental images and sends them to the first large language model;

[0014] The first large language model uses an image encoder to extract image features and focus on environmental features to generate an image feature vector;

[0015] The image feature vector and image text description are aligned using a multimodal alignment and fusion method. The cross-attention mechanism and graph neural network are used to fuse features and generate fused multimodal features.

[0016] The multimodal features are input into the classifier, which outputs the environment type. The environment description is then generated by combining the image and text descriptions.

[0017] Specifically, the step of acquiring user commands and environment data, parsing the user commands and data based on the second language model, and generating command semantics by combining the environment description includes the following steps:

[0018] The second language model receives the user instructions and the environmental data collected by the sensors, parses and extracts the command to be executed, and the user instructions include text instructions and voice instructions;

[0019] Based on the environment description, the risk of the command to be executed is determined using the second major language model;

[0020] Extract valid commands, sort the valid commands, and generate the command semantics.

[0021] Specifically, the second large language model is the ERNIE Speed ​​Large Language Model, and the pre-training process of the second large language model includes:

[0022] A question-and-answer dataset is constructed and preprocessed, the dataset including several dialogue instances involving multiple UAV synchronization and coordination commands;

[0023] The second largest language model was tuned using supervised fine-tuning methods, which involved changing model parameters, adjusting model architecture, and training strategies.

[0024] The visual editor and instruction encoder are integrated into the second large language model, the output layer architecture is optimized, and the pre-training is performed using a mixed precision and distributed training method.

[0025] Specifically, the visual editor adopts a lightweight ViT, utilizes the ViT-Hybrid architecture and LiDAR point cloud processing to achieve multimodal extraction of 3D obstacle features, and stitches them with visual features to achieve multimodal fusion.

[0026] Specifically, the steps of acquiring a preset formation image, calling the first large language model to parse the preset formation image, generating formation instructions based on the command semantics, and controlling the UAV array to complete cluster formation according to the formation instructions include:

[0027] Obtain the preset formation image;

[0028] The first large language model is invoked to parse the preset formation image and extract the preset formation coordinates;

[0029] By integrating the preset formation coordinates and the command semantics, a formation command is generated and sent to the lead drone;

[0030] The lead drone controls the drone array to complete cluster formation and reports back the actual formation coordinates of any drone.

[0031] A second aspect of this application is to provide a drone swarm formation control system based on a multimodal large language model, comprising:

[0032] The environment perception module is used to acquire environmental images captured or received by the UAV array, and generate an environment description by parsing the environmental images through the first language model;

[0033] The command extraction module is used to acquire user commands and environment data, parse the user commands and environment data based on the second language model, and generate command semantics by combining the environment description.

[0034] The response generation module is used to acquire a preset formation image, call the first large language model to parse the preset formation image, combine the command semantics to generate formation instructions, and control the UAV array to complete cluster formation according to the formation instructions.

[0035] A third aspect of this application is to provide a storage medium, which is a non-transitory computer-readable storage medium, storing a program for a UAV swarm formation control method based on a multimodal large language model, wherein the program, when executed by a processor, performs the following steps:

[0036] Acquire environmental images captured or received by a drone array, and generate an environmental description by parsing the environmental images using a first language model;

[0037] The system acquires user commands and environment data, parses the user commands and environment data based on the second language model, and generates command semantics by combining the environment description.

[0038] A preset formation image is obtained, the first large language model is called to parse the preset formation image, and the formation instructions are generated by combining the command semantics. The UAV array is then controlled to complete the cluster formation according to the formation instructions.

[0039] Compared to existing technologies, this application offers the following key advantages: It provides a control method for UAV swarm formation based on a multimodal large language model, comprising: acquiring environmental images captured or received by the UAV array; parsing the environmental images using a first large language model to generate an environmental description; acquiring user commands and environmental data; parsing the user commands and environmental data using a second large language model; and generating command semantics by combining the environmental description; acquiring a preset formation image; calling the first large language model to parse the preset formation image; generating formation commands by combining the command semantics; and controlling the UAV array to complete swarm formation according to the formation commands. This application, through a multimodal large language model and a supervised-trained large language model, can not only analyze the environment but also accurately identify valid commands, eliminate risky commands, and recognize image formation commands, providing crucial technical support for the efficient operation of large-scale UAV swarms and verifying the effectiveness of large language models in multi-agent visual perception and collaboration. This application solves the problem of real-time UAV swarm control based on multimodal image acquisition and recognition. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating a control method for unmanned aerial vehicle (UAV) swarm formation based on a multimodal large language model, as an embodiment of this application.

[0042] Figure 2 This is a flowchart illustrating environmental identification as an embodiment of this application.

[0043] Figure 3 This is a flowchart illustrating the extraction of command semantics in an embodiment of this application.

[0044] Figure 4 This is a flowchart illustrating the pre-training process of the second largest language model based on the ERNIE Speed ​​Large Language Model in this application.

[0045] Figure 5 This is a flowchart illustrating the process of forming drone formations based on a preset formation image, as described in this application.

[0046] Figure 6 This is a schematic diagram of a control system for a drone swarm formation based on a multimodal large language model, which is an embodiment of this application.

[0047] Figure 7 This is a logical architecture diagram of a control method for UAV swarm formation based on a multimodal large language model, which is an embodiment of this application. Detailed Implementation

[0048] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0049] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0050] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0051] like Figure 1 As shown, the first aspect of this application is to provide a control method for UAV swarm formation based on a multimodal large language model, comprising the following steps:

[0052] S1, acquire environmental images captured or received by the UAV array, and generate an environmental description by parsing the environmental images using the first language model;

[0053] S2, acquire user commands and environment data, parse the user commands and environment data based on the second language model, and generate command semantics by combining the environment description;

[0054] S3, acquire a preset formation image, call the first large language model to parse the preset formation image, combine the command semantics to generate formation instructions, and control the UAV array to complete cluster formation according to the formation instructions.

[0055] Specifically, the drone array includes a lead drone and several follower drones. The lead drone is used to capture or receive environmental images and the preset formation images. The lead drone and any of the follower drones are equipped with several sensors to acquire environmental data. The number of follower drones exceeds one. Each drone in the array is assigned a number; for example, the lead drone is numbered 1, and the follower drones are numbered sequentially starting from 2. User commands, including drone numbers, can be set according to the prompt word engineering. For example, if the user command is "first drone moves forward 5 meters," the parsed command semantics are "(1, forward, 5)," where "1" represents the drone number, "forward" is the command executed by the drone, and "5" represents the distance in meters.

[0056] In scenarios involving multiple unmanned aerial vehicles (UAVs) swarms, all participating UAVs are connected to the same local network environment. The main control computer in this network environment is the command and dispatch center, ensuring efficient and precise control and management of the UAV swarm.

[0057] Specifically, in step S1, the environment description includes environment type and environment features. For example, if a sofa is detected in the environment image, the generated environment description may include "the environment type is a living room, and the space is small". Or, if trees are detected in the environment image, the generated environment description may be "the environment type is a forest, and the space is large, with many obstacles".

[0058] Specifically, the first large language model employs a multimodal large language model (MLLM) capable of real-time image processing. Preferably, the first large language model in this application uses OpenAI's GPT-4 and embeds an image encoder, enabling it to combine computer vision and natural language processing techniques to align and fuse visual information and textual semantics, achieving accurate classification and description of the environment. During the pre-training process of the first large language model, this application prepared 1000 images of different scene types and set several environment description instructions. By extracting environmental feature semantics, accurate identification of environment types and output of corresponding environment descriptions are achieved.

[0059] Specifically, such as Figure 2 As shown, step S1 involves acquiring user commands and environmental images captured or received by the UAV array, and analyzing the environmental images using a first language model to obtain environmental information. This includes the following steps:

[0060] S11, the pilot drone captures or receives the environmental images and sends them to the first large language model;

[0061] S12, the first large language model uses an image encoder to extract image features and focus on environmental features to generate an image feature vector;

[0062] S13, the image feature vector and the image text description are aligned through a multimodal alignment and fusion method, and the fused multimodal features are generated by using a cross-attention mechanism and a graph neural network for feature fusion.

[0063] S14, input the multimodal features into the classifier, output the environment type, and combine the image text description to generate the environment description.

[0064] Specifically, the drone can directly use its camera to capture environmental images of its surroundings, or it can receive environmental images sent by the command and dispatch center via a local network. The resolution of the environmental images is random, but preferably 1024×1024 RGB images. Simultaneously, the drone is equipped with several recording sensors to collect information such as its latitude, longitude, and altitude.

[0065] In step S11, the pilot drone sends the captured environmental images to the first large language model. The first large language model uses OpenAI's GPT-4 and embeds an image encoder. In step S12, after receiving the environmental images, the image encoder is first used to extract image features and focus on environmental features, generating image feature vectors. Specifically, the image encoder (Visual Encoder) includes a Vision Transformer (ViT) architecture and a CNN architecture. The former divides the image into patches and models global contextual relationships through a self-attention mechanism, such as distinguishing the dense distribution of trees in a "forest" from the sparse layout of furniture in a "bedroom." The latter uses architectures such as ResNet, VGG, and EfficientNet, extracting hierarchical features through multi-layer convolutions, including edges, textures, object parts, and the overall scene. Image features are extracted through the image encoder, and then backbone network optimization and data augmentation are used to focus on environmental features. Backbone network optimization is designed for environmental classification tasks, training the network to focus on scene-related features, such as the bed and windows in the bedroom, and the tree trunks and shrubs in the forest. Data augmentation uses rotation, scaling, and color adjustment to simulate real-world lighting changes, improving robustness.

[0066] In step S13, the image feature vector and the image text description are aligned using a multimodal alignment and fusion method. A cross-attention mechanism and a graph neural network are used for feature fusion to generate fused multimodal features. Specifically, a projection layer unifies the image feature vector and the text token embedding vector to the same dimension. Then, through contrastive learning, InfoNCE Loss is used to allow the first language model to learn that the image features corresponding to a certain environment category are similar to the embeddings of environmental features, while maximizing the differences from the embeddings of other environment categories. Furthermore, a contextual fusion mechanism is employed, allowing attention to key regions in the image through the cross-attention mechanism. For example, when identifying the environment category as "bedroom," the focus is on "bed." Simultaneously, a graph neural network (GNN) is used to construct a relationship graph between the image and regions, such as occlusion between objects and spatial layout.

[0067] In step S14, the multimodal features are input into the classifier, which outputs the environment type and combines it with the image text description to generate the environment description.

[0068] Specifically, the classifier includes a fully connected layer and a label alignment layer. The fully connected layer inputs the fused multimodal features into the classifier and outputs the environment category probability, such as bedroom, lawn, beach, etc. The label alignment layer establishes a mapping between image features and image text labels through label data. Through step S14, the final output is the environment type and environment description, such as, "The drone array is located in a bedroom, with furniture such as a bed and wardrobe, an area of ​​15 square meters, a height of 2.8 meters, and a small range of movement." Here, "bedroom" is the environment type, "bed" and "wardrobe" have corresponding image text labels in the first language model, the area and height can be measured in real time by the drone's built-in sensors, and "small range of movement" is the descriptive text output by the first language model after inference. In the database of the first language model, there are a large number of mapping relationships between environment types and image labels, as well as mapping relationships between environment types and corresponding movement range restrictions. During the training process of the first language model, hundreds of environment images of different environment types were used, and each environment type contains several feature objects that can reflect the environment type. Therefore, the trained first large language model can identify and describe various environmental types. Thus, the UAV swarm formation method based on the large language model proposed in this application can improve the comprehensive understanding and awareness of the surrounding environment of UAV swarms or arrays, providing strong support for real-time control of UAV swarms in complex scenarios. Steps S11-S14 can be understood as the pre-training process of the first large language model, continuously optimizing its recognition accuracy and precision through the identification of numerous images of known environmental types.

[0069] After identifying the environment type, this application further extracts and analyzes user instructions to generate precise command semantics. For example... Figure 3 As shown, step S2, which involves acquiring user commands and environment data, parsing the user commands and environment data based on a pre-trained second language model, and generating command semantics by combining the environment description, includes the following steps:

[0070] S21, the user instruction and the environmental data collected by the sensor are received through the pre-trained second language model, and the command to be executed is parsed and extracted. The user instruction includes text instruction and voice instruction.

[0071] S22, Based on the environment description, determine the risk of the command to be executed using the second language model;

[0072] S23, extract valid commands, sort the valid commands, and generate the command semantics.

[0073] Specifically, the second major language model is either the ERNIE Speed ​​large language model developed by Baidu, or a large language model capable of judging the risk of user commands and accurately extracting valid, risk-free commands. In this application, the second major language model based on the ERNIE Speed ​​large language model is first pre-trained, such as... Figure 4 As shown, the pre-training steps include:

[0074] S2-1, Construct a question-and-answer dataset and perform data preprocessing. The question-and-answer dataset includes 20,000 dialogue instances involving multiple UAV synchronization and coordination commands.

[0075] S2-2, using supervised fine-tuning methods to adjust the second largest language model, changing model parameters, adjusting model architecture and training strategy;

[0076] S2-3, integrate the visual editor and instruction encoder into the second large language model, optimize the output layer architecture, and use mixed precision and distributed training methods for pre-training.

[0077] Specifically, the question-answering dataset includes 20,000 dialogue instances involving multiple drone synchronization and coordination commands, such as: "The drone swarm is now in a factory, operating at a medium range. The first drone avoids the obstacle in front. The second drone has a hazard below; avoid it. The third drone rises sharply." and "The drone swarm is now in a bedroom, operating at a small range. The first drone moves back half a meter. The second drone rises 100 centimeters. The third drone moves back slightly." These involve multiple drone synchronization and coordination commands. Using this dataset, a pre-trained large-scale language model specifically designed to understand and parse human intentions was trained. This model can understand direct or ambiguous user instructions and extract precise control commands from them.

[0078] Specifically, during pre-training, the second language model's input consists of user commands and environmental data collected by the drone's sensors, as well as the environmental description output by the first language model. User commands can be text-based or voice-based, such as "The first drone moves forward a little" or "Form a triangle." The environmental data collected by the drone's sensors includes the drone's latitude and longitude, altitude, relative altitude within its environment, ambient temperature, weather, and other environmental information.

[0079] Step S2-2 involves using supervised fine-tuning to adjust the second language model, changing model parameters, adjusting the model architecture and training strategy, specifically including:

[0080] Data preprocessing: converting natural language commands into specific control parameters, such as flight paths and swarm coordination commands;

[0081] Model adaptation: The architecture or training strategy of the lightweight original ERNIESpeed ​​model is adjusted through methods such as knowledge distillation and mixed precision training to adapt to the real-time and low-latency requirements of UAV control.

[0082] In data preprocessing, text enhancement and scene enhancement are implemented to further improve the accuracy of extracted instructions, such as synonym replacement, changing "maintain a 5-meter distance" to "maintain a safe distance of 5 meters". In another embodiment, weather simulation and dynamic obstacle injection are implemented to improve the ability to assess environmental risks.

[0083] Specifically, the knowledge distillation design includes a teacher model and a student model. The teacher model is pre-trained on ERNIE 3.0-Large (175B parameters) and its output includes text understanding (instruction intent classification, such as formation / obstacle avoidance) and visual reasoning, such as scene understanding features. The student model is based on ERNIE Speed-Drone (1.1B parameters), and its distillation strategy includes feature matching and corresponding alignment.

[0084] Specifically, hybrid precision training includes precision configuration and a parallel strategy. Precision configuration includes a learning rate of 3e-5, 4 iterations, and a training loss of approximately 0.124 after about 3 hours of training.

[0085] Steps S2-3 involve integrating the visual editor and instruction encoder into the second large language model, optimizing the output layer architecture, and performing pre-training using a mixed precision and distributed training method, including:

[0086] Multimodal processing: Combine the environment description output by the first major language model with the visual editor and instruction encoder for multimodal training;

[0087] Evaluation and optimization: Design evaluation metrics suitable for UAV control tasks, such as command accuracy and response time.

[0088] Specifically, the visual editor adopts a lightweight ViT, utilizes the ViT-Hybrid architecture and LiDAR point cloud processing to achieve multimodal extraction of 3D obstacle features, and stitches them with visual features to achieve multimodal fusion.

[0089] ViT reduces computational complexity and improves real-time control through a lightweight design, and is adaptable to drone scenarios, integrating with the image encoder of the leading language model. The hybrid encoder employs a lightweight CNN with 3×3 convolutions (stride 2) and pooling to extract local features, and uses multi-head self-attention to achieve layer normalization and fusion. LiDAR point cloud processing takes 3D point clouds as input, performs neighborhood sampling at the local feature layer to construct a local connectivity graph, aggregates the global topology at the global feature layer, restores point cloud resolution through upsampling to enhance details, and finally outputs a dense 256-dimensional feature vector to express the overall semantics.

[0090] During the visual feature concatenation process, the environment description output by the first large language model is combined with the overall semantics from the second large language model, and then fused with the instructions output by the instruction encoder. Feature alignment and dimensional concatenation are performed to extract precise and effective instructions from ambiguous user commands. In actual UAV swarm control, user commands may be ambiguous. Through the above pre-training process, the second large language model can extract precise control command sequences from ambiguous user commands. However, control command sequences may contain risky situations. For example, the environment description output by process S1 is "the UAV swarm is currently in a bedroom, with a small operating range," while the user command is "move forward 60 meters as a whole." The pre-trained large language model analyzes the user command and uses the UAV sensors to obtain the length, width, and height of the bedroom, as well as the distance of each UAV from the bedroom wall. Through reasoning, it derives risk-free commands such as "move as a whole" and "move forward synchronously," as well as risky commands such as "move 60 meters." For risky commands, the second large language model outputs risk warning information to the computer.

[0091] Specifically, such as Figure 5 As shown, step S3, which involves acquiring a preset formation image, calling the first large language model to parse the preset formation image, generating formation instructions based on the command semantics, and controlling the UAV array to complete cluster formation according to the formation instructions, includes the following steps:

[0092] S31, Obtain the preset formation image;

[0093] S32, call the first large language model to parse the preset formation image and extract the preset formation coordinates;

[0094] S33, integrate the preset formation coordinates and the command semantics to generate formation instructions and send them to the lead drone;

[0095] S34, the lead drone controls the drone array to complete the cluster formation and feeds back the actual formation coordinates of any drone.

[0096] The preset formation image can be sent from a computer or captured by the camera of the lead drone. For example, a user shows the lead drone a preset formation image, which the lead drone captures in real time and sends to the first language model. The first language model parses the preset formation image and extracts formation commands, including preset formation coordinates. Then, it fuses the preset formation coordinates with the command semantics output by the second language model to generate valid formation commands, which are sent to the lead drone, and then forwarded to the other follower drones. During this process, the range of the preset formation coordinates must be inferred and must not exceed a safe range.

[0097] Optionally, the preset formation image can be a triangle, rectangle, pentagon, irregular shape, etc. The formation is done from the outside in according to the number of drones in the array. For example, for a drone array containing 4 drones, if a triangular formation is required, the lead drone and 2 follower drones are first arranged at the three vertices of the triangle, and the fourth drone is arranged within the triangle.

[0098] Specifically, the formation strategy in this application adopts a leader-follower approach. Depending on the formation specifications, the required geometric formation shape can be determined by the absolute position of the UAVs. Relative displacement ,distance ,position and interior angles The absolute position of the UAVs is described in this application, taking into account that the positioning system in the sensor can provide the absolute position of any UAV. Therefore, the formation control law for each UAV can be designed as formula (1).

[0099] (1)

[0100] in, It is a drone The three-dimensional movement speed is also the control command given to the drone. V This represents an index set containing all drones. It is a gain parameter used to adjust the convergence speed. The drone's real-time location is obtained from the positioning map. It is the expected position decomposed from the expected array and obtained from the output of the large language model.

[0101] Specifically, such as Figure 7The diagram illustrates the logical architecture of a UAV swarm formation control method based on a multimodal large language model, according to an embodiment of this application. The method involves acquiring environmental images captured or received by the UAV array, parsing these images using a first large language model to generate an environmental description, acquiring environmental data through UAV sensors, obtaining user commands from the user, inputting the acquired user commands, environmental data, and environmental description into a second large language model, parsing the user commands and environmental data, and generating command semantics based on the environmental description, including command determination and type. Then, a preset formation image is acquired, and the first large language model is invoked to parse the preset formation image, extracting coordinates from specific commands and transmitting them to the second large language model. The second large language model combines the command semantics to generate formation commands, image understanding commands, and sensor activation commands. Based on the formation commands, the UAV array is controlled to complete swarm formation. Image understanding information is output by the first large language model through the image understanding commands, and sensor information is output by the sensors. During the formation process, the final response and feedback are divided into three categories: action response of the UAV array, image feedback, and sensor data feedback for each category of command. The feedback information or data is returned to the command and dispatch center via the lead UAV.

[0102] Specifically, such as Figure 6 As shown, a second aspect of this application is to provide a control system 1 for unmanned aerial vehicle (UAV) swarm formation based on a multimodal large language model, comprising:

[0103] The environment perception module 10 is used to acquire environmental images captured or received by the UAV array, and generate an environment description by parsing the environmental images through the first language model.

[0104] Command extraction module 20 is used to acquire user instructions and environment data, parse the user instructions and the data based on a pre-trained second language model, and generate command semantics by combining the environment description.

[0105] The response generation module 30 is used to acquire a preset formation image, call the first large language model to parse the preset formation image, combine the command semantics to generate formation instructions, and control the UAV array to complete cluster formation according to the formation instructions.

[0106] Database module 40 is used to store question-and-answer datasets;

[0107] The drone array 50 includes a lead drone and several follower drones. The lead drone is used to capture or receive environmental images and the preset formation images, and sends formation instructions from the command and dispatch center to any of the follower drones, and reports the actual formation position of any of the follower drones back to the command and dispatch center. The lead drone and any of the follower drones are equipped with several sensors to acquire the environmental data.

[0108] A third aspect of this application provides a storage medium, which is a non-transitory computer-readable storage medium, storing a program for a control method of UAV swarm formation based on a multimodal large language model, wherein the program, when executed by a processor, implements the following steps:

[0109] Acquire environmental images captured or received by a drone array, and generate an environmental description by parsing the environmental images using a first language model;

[0110] The system acquires user commands and environment data, parses the user commands and data based on a pre-trained second language model, and generates command semantics by combining the environment description.

[0111] A preset formation image is obtained, the first large language model is called to parse the preset formation image, and the formation instructions are generated by combining the command semantics. The UAV array is then controlled to complete the cluster formation according to the formation instructions.

[0112] The following specific embodiments further describe a control method for UAV swarm formation based on a multimodal large language model.

[0113] In one embodiment, a drone array is set up, including a lead drone numbered "1" and 20 follower drones numbered 2-21. Both the lead and follower drones are equipped with sensors to collect information such as position and distance. The drone swarm control method is tested in factory and lawn environments. Thirty user commands are set up, such as "The current image in front of the drone shows an object at a distance of approximately 100 cm. Based on this information, please describe the type of environment in which the drone is located and its movable range, and extract environmental information."

[0114] The drones collect environmental information through onboard sensors and cameras, then use the GPT-4 image understanding model to interpret this information. Specifically, the GPT-4 model retrieves data such as the distance from the lead drone to obstacles ahead, images captured by the cameras, and the specific position and altitude of each drone. Before command extraction, the formation system inputs real-time images captured by the drones and pre-set language prompts into the GPT-4 model for environmental analysis. For example, some language prompts might say, "The current image in front of the drone shows an object approximately 100 cm away. Based on this information, please describe the type of environment the drone is in and its movable range, and extract the environmental information." Based on these specific language prompts, the multimodal large language model GPT-4 returns a structured environmental description and assesses the movable range, such as, "The drone is currently in the bedroom, and its movable range is small." Subsequently, the pre-trained large language model integrates this environmental information and user commands, then evaluates the safety of the current user command based on the environmental information; unsafe commands are rejected. Based on different user commands, sensor feedback, and environmental information, the model re-evaluates the risk level of the command, resulting in different command outputs. The multimodal GPT-4 and the pre-trained ERNIESpeed ​​large language model play a decisive role in the command extraction module.

[0115] For a pre-trained large language model, the input consists of relevant user commands and environmental information surrounding the drone, while the output is the precise command extracted by the large language model, such as "(1, forward, 50)", where "1" represents the drone's sequence number, "forward" is the command executed by the drone, and "50" represents the distance. By accurately extracting and parsing natural language commands using a pre-trained large language model, the ultimate goal is to achieve real-time, effective, and end-to-end control of drone swarm formations in several different environments.

[0116] In this embodiment, a self-instruction method was used to systematically design and generate approximately 600 control command sequences for UAV swarms. These sequences included environmental descriptions and simulated user input text commands. This method involves inputting specific formatted prompts into GPT-4, which then generates multiple datasets in batches. During data generation, data generation similarity can be set to filter and expand the datasets. These commands were used in conjunction with a pre-trained ERNIESpeed ​​large language model to evaluate its understanding and accurate execution of complex formation control commands. These data were also categorized into four types: safe command sets, dangerous command sets, mixed command sets, and non-existent command sets. Through statistical analysis of the parsing and response results of each command, this application calculated that the average accuracy of the large language model in handling such high-level and coordinated tasks reached 82.7%. The calculation method is shown in formula (2), and the specific statistical data is shown in Table 1.

[0117] (2)

[0118] Table 1

[0119]

[0120] To test the image processing capabilities of the multimodal GPT-4, this application selected dozens of examples of conventional geometric shapes and irregular graphics. These pre-defined array images were input one by one into the GPT-4 large language model to test the model's ability to accurately extract spatial coordinate information from image input (visual input). After a series of experimental operations, this application also quantitatively evaluated the model's performance in this image recognition and coordinate localization task and calculated the corresponding accuracy, which reached 83.8%. The accuracy calculation method is shown in formula (3).

[0121] (3)

[0122] Table 2

[0123]

[0124] To ensure proper operation of formation control within the UAVs, and to utilize the UAV swarm formation method combining the two large language models presented in this application, the flight accuracy of the UAV array consisting of the aforementioned 21 UAVs was tested in different environmental types, such as a forklift and a lawn. The test results are shown in Table 2, where the slight difference between the target position and the actual hovering position demonstrates the high accuracy of the UAV flight.

[0125] In summary, this application presents a control method for UAV swarm formation based on a multimodal large language model, providing an interactive control approach for UAV swarms. It employs a multimodal large language model to analyze image data captured by the UAV swarm, deeply integrating user language or text input, environmental recognition information, and the real-time status of the UAV swarm. This allows for accurate extraction of appropriate control commands based on the user's natural language instructions, achieving smooth interaction with human users. When receiving ambiguous or potentially dangerous human commands, the pre-trained large-scale language model utilizes common-sense reasoning and dataset learning to accurately identify user intentions and filter out valid operational commands, thus rejecting potentially risky actions. The average command extraction accuracy reaches 82.7%. Specifically, this application uses the multimodal GPT-4 large language model to process images captured by the lead UAV group, performing swarm formation control based on image features. By acquiring the corresponding coordinates and changing the UAV swarm formation configuration, an accuracy of 83.8% was ultimately achieved, effectively advancing visual feature-based formation control technology. This breakthrough not only verifies the enormous potential of large-scale language models in multi-agent visual perception and collaboration but also provides crucial technical support for the efficient operation of large-scale UAV swarms. In summary, this study proposes a practical and feasible application framework and training method, demonstrating the importance of large language models in enhancing the contextual understanding, dynamic adaptability, and task execution efficiency of swarm UAV systems.

[0126] In summary, this application provides a control method for UAV swarm formation based on a multimodal large language model, comprising: acquiring environmental images captured or received by the UAV array; parsing the environmental images using a first large language model to generate an environmental description; acquiring user commands and environmental data; parsing the user commands and data based on a pre-trained second large language model; generating command semantics by combining the environmental description; acquiring a preset formation image; calling the first large language model to parse the preset formation image; generating formation commands by combining the command semantics; and controlling the UAV array to complete swarm formation according to the formation commands. This application, through a multimodal large language model and a supervised-trained large language model, can not only analyze the environment but also accurately identify valid commands, eliminate risky commands, and identify image formation commands, providing key technical support for the efficient operation of large-scale UAV swarms and verifying the effectiveness of large language models in multi-agent visual perception and collaboration. This application solves the problem of real-time UAV swarm control based on multimodal image acquisition and recognition.

[0127] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions are not in essence a departure from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for controlling UAV swarm formation based on a multimodal large language model, characterized in that, Includes the following steps: Acquire environmental images captured or received by a drone array, and generate an environmental description by parsing the environmental images using a first language model; The system acquires user commands and environment data, parses the user commands and environment data based on the second language model, and generates command semantics by combining the environment description. Obtain a preset formation image, call the first large language model to parse the preset formation image, combine the command semantics to generate formation instructions, and control the UAV array to complete cluster formation according to the formation instructions; The drone array includes at least one lead drone and several follower drones; the lead drone is used to capture or receive the environmental images and the preset formation images; the lead drone and any of the follower drones are equipped with several sensors to acquire the environmental data; The process of acquiring environmental images captured or received by a drone array and generating an environmental description by parsing the environmental images using a first language model includes the following steps: The pilot drone captures or receives the environmental images and sends them to the first large language model; The first large language model uses an image encoder to extract image features and focus on environmental features to generate an image feature vector; The image feature vector and image text description are aligned using a multimodal alignment and fusion method. The cross-attention mechanism and graph neural network are used to fuse features and generate fused multimodal features. The multimodal features are input into a classifier, which outputs an environment type. The environment description is then generated by combining the image and text descriptions. The process of acquiring user commands and environment data, based on a second major language model, parsing the user commands and environment data, and generating command semantics by combining the environment description includes the following steps: The second language model receives the user instructions and the environmental data collected by the sensors, parses and extracts the commands to be executed, and the user instructions include text instructions and voice instructions; Based on the environment description, the risk of the command to be executed is determined using the second major language model; Extract valid commands, sort the valid commands, and generate the command semantics.

2. The UAV swarm formation control method based on a multimodal large language model according to claim 1, characterized in that, The first large language model is the multimodal GPT-4 large language model.

3. The UAV swarm formation control method based on a multimodal large language model according to claim 1, characterized in that, The second largest language model is the ERNIE Speed ​​Large Language Model. The pre-training process of the second largest language model includes: A question-and-answer dataset is constructed and preprocessed, the dataset including several dialogue instances involving multiple UAV synchronization and coordination commands; The second largest language model was tuned using supervised fine-tuning methods, which involved changing model parameters, adjusting model architecture, and training strategies. The visual editor and instruction encoder are integrated into the second large language model, the output layer architecture is optimized, and the pre-training is performed using a mixed precision and distributed training method.

4. The UAV swarm formation control method based on a multimodal large language model according to claim 3, characterized in that, The visual editor uses a lightweight ViT architecture and LiDAR point cloud processing to extract 3D obstacle features in a multimodal manner, which are then stitched together with visual features to achieve multimodal fusion.

5. The UAV swarm formation control method based on a multimodal large language model according to claim 1, characterized in that, The steps of acquiring a preset formation image, calling the first large language model to parse the preset formation image, generating formation instructions based on the command semantics, and controlling the UAV array to complete cluster formation according to the formation instructions include: Obtain the preset formation image; The first large language model is invoked to parse the preset formation image and extract the preset formation coordinates; By integrating the preset formation coordinates and the command semantics, a formation command is generated and sent to the lead drone; The lead drone controls the drone array to complete cluster formation and reports back the actual formation coordinates of any drone.

6. A UAV swarm formation control system based on a multimodal large language model, to implement the steps of the UAV swarm formation control method based on a multimodal large language model as described in claim 1, characterized in that, include: The environment perception module is used to acquire environmental images captured or received by the UAV array, and generate an environment description by parsing the environmental images through the first language model; The command extraction module is used to acquire user commands and environment data, parse the user commands and environment data based on the second language model, and generate command semantics by combining the environment description. The response generation module is used to acquire a preset formation image, call the first large language model to parse the preset formation image, combine the command semantics to generate formation instructions, and control the UAV array to complete cluster formation according to the formation instructions.

7. A storage medium, characterized in that, The storage medium is a non-transitory computer-readable storage medium storing a program for a UAV swarm formation control method based on a multimodal large language model as described in any one of claims 1-5. When the program is executed by a processor, it implements the following steps: Acquire environmental images captured or received by a drone array, and generate an environmental description by parsing the environmental images using a first language model; The system acquires user commands and environment data, parses the user commands and environment data based on the second language model, and generates command semantics by combining the environment description. A preset formation image is obtained, the first large language model is called to parse the preset formation image, and the formation instructions are generated by combining the command semantics. The UAV array is then controlled to complete the cluster formation according to the formation instructions.

Images