Unmanned aerial vehicle cluster formation control method and system based on multi-modal large language model

Abstract

The invention provides an unmanned aerial vehicle cluster formation control method and system based on a multi-mode large language model, and the method comprises the steps: obtaining an environment image shot or received by an unmanned aerial vehicle array, and analyzing the environment image through a first large language model to generate an environment description; acquiring a user instruction and environment data, analyzing the user instruction and the environment data based on a second large language model, and generating command semantics in combination with the environment description; and acquiring a preset formation image, calling the first large language model to analyze the preset formation image, generating a formation instruction in combination with the command semantics, and controlling the unmanned aerial vehicle array to complete cluster formation according to the formation instruction. According to the invention, through the multi-modal large language model and the supervised and trained large language model, real-time unmanned aerial vehicle cluster control based on multi-modal image acquisition and identification is realized.

Description

Technical Field

[0001] The present application relates to the field of drone control technology, and in particular to a drone cluster formation control method and system based on a multimodal large language model. Background Art

[0002] Collaborative unmanned aerial vehicle (UAV) systems are playing an increasingly important role in various fields due to their high flexibility, integrated intelligent functions, and comprehensive collaborative capabilities. By integrating high-precision sensors, efficient wireless communication technologies, and advanced autonomous decision-making algorithms, these systems can effectively coordinate and optimize various tasks across multiple UAVs in complex environments. Existing research has utilized large language models (LLMs) to achieve formation control of UAVs. However, existing LLM applications for UAV formation control mostly focus on conversational interaction. For example, invention patent application number 202311642952.9 discloses a method and system for precise command and control of UAV swarms based on LLMs. This method achieves control of UAV swarms by converting user-input voice / text data into a vertical LLM that can be input into the command and control domain. However, this method cannot achieve formation control using images. Existing applications of LLM-based UAV swarm control mostly focus on conversational interaction rather than multimodal image perception and understanding, failing to achieve real-time UAV swarm control based on multimodal image acquisition and recognition.

[0003] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the existing technology, the present application provides a method and system for drone cluster formation control based on a multimodal large language model, aiming to solve the problem in the existing technology of lack of drone cluster control based on a large language model of multimodal image perception and understanding, and to realize real-time drone cluster control based on multimodal image acquisition and recognition.

[0005] This application is achieved through the following technical solutions: The first aspect of the present application is to provide a method for controlling a swarm of drones based on a multimodal large language model, comprising the following steps: Obtaining an environment image captured or received by the drone array, and parsing the environment image using the first language model to generate an environment description; Obtaining user instructions and environment data, parsing the user instructions and the environment data based on the second language model, and generating command semantics in combination with the environment description; Obtain a preset formation image, call the first language model to parse the preset formation image, generate a formation instruction based on the command semantics, and control the drone array to complete the cluster formation according to the formation instruction.

[0006] Specifically, the drone array includes at least one pilot drone and several follower drones; the pilot drone is used to capture or receive the environmental image and the preset formation image; the pilot drone and any of the follower drones are equipped with several sensors for acquiring the environmental data.

[0007] Specifically, the first large language model is the multimodal GPT-4 large language model.

[0008] Specifically, the step of acquiring an environment image captured or received by an array of drones and parsing the environment image using a first language model to generate an environment description includes the following steps: The pilot drone captures or receives the environment image and sends it to the first language model; The first language model uses an image encoder to extract image features and focus on environmental features to generate an image feature vector; The image feature vector and the image text description are aligned through a multimodal alignment and fusion method, and the cross-attention mechanism and graph neural network are used to perform feature fusion to generate fused multimodal features; The multimodal features are input into a classifier, the environment type is output, and the environment description is generated by combining the image text description.

[0009] Specifically, the acquiring of user instructions and environment data, parsing the user instructions and the data based on the second language model, and generating command semantics in combination with the environment description includes the following steps: receiving the user instruction and the environmental data collected by the sensor through the second largest language model, parsing and extracting the command to be executed, wherein the user instruction includes a text instruction and a voice instruction; Based on the environment description, determining the risk of the command to be executed by using the second largest language model; Valid commands are extracted, sorted, and the command semantics are generated.

[0010] Specifically, the second largest language model is the ERNIE Speed large language model, and the pre-training process of the second largest language model includes: Constructing a question-answering dataset and performing data preprocessing, wherein the question-answering dataset includes several dialogue instances involving synchronized coordinated commands of multiple drones; Use supervised fine-tuning methods to adjust the second language model, change model parameters, adjust model architecture and training strategy; The visual editor and command encoder are integrated into the second language model, the output layer architecture is optimized, and the pre-training is performed using mixed precision and distributed training methods.

[0011] Specifically, the visual editor adopts lightweight ViT, utilizes ViT-Hybrid architecture and LiDAR point cloud processing to realize multimodal extraction of three-dimensional obstacle features, and splices them with visual features to achieve multimodal fusion.

[0012] Specifically, the method of obtaining a preset formation image, calling the first language model to parse the preset formation image, generating a formation instruction in combination with the command semantics, and controlling the UAV array to complete the cluster formation according to the formation instruction includes the following steps: Acquiring the preset formation image; calling the first language model to parse the preset formation image and extract the preset formation coordinates; Fusion of the preset formation coordinates and the command semantics to generate a formation instruction and send it to the pilot drone; The pilot drone controls the drone array to complete the cluster formation and feeds back the actual formation coordinates of any drone.

[0013] The second aspect of the present application is to provide a UAV swarm formation control system based on a multimodal large language model, comprising: An environment perception module is used to obtain an environment image captured or received by the drone array, and parse the environment image using the first language model to generate an environment description; A command extraction module is used to obtain user instructions and environment data, parse the user instructions and the environment data based on the second language model, and generate command semantics in combination with the environment description; The response generation module is used to obtain a preset formation image, call the first language model to parse the preset formation image, generate a formation instruction based on the command semantics, and control the drone array to complete the cluster formation according to the formation instruction.

[0014] A third aspect of the present application is to provide a storage medium, which is a non-transitory computer-readable storage medium and stores a program for a method for controlling a swarm of drones based on a multimodal large language model. When the program is executed by a processor, the following steps are implemented: Obtaining an environment image captured or received by the drone array, and parsing the environment image using the first language model to generate an environment description; Obtaining user instructions and environment data, parsing the user instructions and the environment data based on the second language model, and generating command semantics in combination with the environment description; Obtain a preset formation image, call the first language model to parse the preset formation image, generate a formation instruction based on the command semantics, and control the drone array to complete the cluster formation according to the formation instruction.

[0015] Compared with the existing technology, the present application mainly has the following beneficial effects: The present application provides a control method for a drone cluster formation based on a multimodal large language model, comprising: obtaining an environmental image taken or received by a drone array, parsing the environmental image through a first large language model to generate an environmental description; obtaining user instructions and environmental data, parsing the user instructions and the environmental data based on a second large language model, and generating command semantics in combination with the environmental description; obtaining a preset formation image, calling the first large language model to parse the preset formation image, generating formation instructions in combination with the command semantics, and controlling the drone array to complete cluster formation according to the formation instructions. The present application uses a multimodal large language model and a supervised large language model to not only analyze the environment, but also accurately identify valid instructions, eliminate risky instructions, and identify image formation instructions, thereby providing key technical support for the efficient operation of large-scale drone groups and verifying the effectiveness of large language models in multi-agent visual perception and collaboration. The present application solves the problem of real-time drone cluster control based on multimodal image acquisition and recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 This is a flowchart of a method for controlling a drone cluster formation based on a multimodal large language model in an embodiment of the present application.

[0018] Figure 2 A flowchart of environment identification according to an embodiment of the present application.

[0019] Figure 3 This is a flowchart of extracting command semantics according to an embodiment of the present application.

[0020] Figure 4 This is a flowchart of the pre-training of the second largest language model based on the ERNIE Speed large language model of this application.

[0021] Figure 5 This is a flowchart of the application for forming drone formations based on preset formation images.

[0022] Figure 6 This is a schematic diagram of a control system for a drone swarm formation based on a multimodal large language model in an embodiment of the present application.

[0023] Figure 7 This is a logical architecture diagram of a control method for a drone cluster formation based on a multimodal large language model in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] It should be understood that the terms used in this specification of the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. As used in the specification of the embodiments of the present application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0027] like Figure 1 As shown, the first aspect of the present application is to provide a control method for a drone cluster formation based on a multimodal large language model, comprising the following steps: S1, obtaining an environment image captured or received by an array of drones, and parsing the environment image using a first language model to generate an environment description; S2, obtaining user instructions and environment data, parsing the user instructions and the environment data based on the second language model, and generating command semantics in combination with the environment description; S3: Obtain a preset formation image, call the first language model to parse the preset formation image, generate a formation instruction based on the command semantics, and control the drone array to complete the cluster formation according to the formation instruction.

[0028] Specifically, the drone array includes a pilot drone and several follower drones; the pilot drone is used to capture or receive the environmental image and the preset formation image; the pilot drone and any of the follower drones are equipped with several sensors for acquiring the environmental data. The number of follower drones exceeds one. In the drone array, each drone is numbered. For example, the pilot drone is numbered 1, and the follower drones are numbered sequentially starting from 2. User instructions including the drone number can be set based on the prompt word project. For example, the user instruction "First drone, move forward 5 meters" is parsed to form the command semantics "(1, forward, 5)", where "1" represents the drone number, "forward" is the command executed by the drone, and "5" is the distance in meters.

[0029] In scenarios where multiple unmanned aerial vehicles (UAVs) form a swarm, all participating UAVs are connected to the same local network environment. The primary control computer in this network environment serves as the command and dispatch center, ensuring efficient and precise control and management of the UAV swarm.

[0030] Specifically, in step S1, the environment description includes the environment type and environment features. For example, if a sofa is recognized in the environment image, the generated environment description may include "the environment type is a living room with a small space", or if trees are recognized in the environment image, the generated environment description may be "the environment type is a forest with a large space and many obstacles".

[0031] Specifically, the first large language model uses a multimodal large language model (MLLM) capable of real-time image processing. Preferably, the first large language model of this application uses OpenAI's GPT-4 and embeds an image encoder, enabling it to combine computer vision and natural language processing technologies to align and fuse visual information and textual semantics, achieving accurate classification and description of the environment. During the pre-training process of the first large language model, this application prepared 1,000 images of environment types in different scenarios and set several environment description instructions. By extracting the semantics of environmental features, accurate identification of environment types and output of corresponding environment descriptions are achieved.

[0032] Specifically, if Figure 2 As shown, step S1, obtaining user instructions and obtaining environmental images taken or received by the drone array, and analyzing the environmental images through the first language model to obtain environmental information, includes the following steps: S11, the pilot drone photographs or receives the environment image and sends it to the first language model; S12, the first language model uses an image encoder to extract image features and perform environmental feature focusing to generate an image feature vector; S13, aligning the image feature vector and the image text description through a multimodal alignment and fusion method, and performing feature fusion using a cross-attention mechanism and a graph neural network to generate fused multimodal features; S14, inputting the multimodal features into a classifier, outputting the environment type, and generating the environment description in combination with the image text description.

[0033] Specifically, the drone can directly capture images of its surroundings using its camera, or it can receive images sent by the command and dispatch center via the local network. The resolution of the images is random, preferably 1024×1024 RGB. The drone is also equipped with several recording sensors that can collect information such as its latitude, longitude, and altitude.

[0034] In step S11, the pilot drone sends the captured environmental image to the first language model. This language model uses OpenAI's GPT-4 embedded in an image encoder. In step S12, after receiving the environmental image, the image encoder first extracts image features and performs environmental feature focus, generating an image feature vector. Specifically, the image encoder (Visual Encoder) comprises a Vision Transformer (ViT) architecture and a CNN architecture. The ViT divides the image into patches and uses a self-attention mechanism to model global contextual relationships, such as distinguishing the dense distribution of trees in a "forest" from the sparse layout of furniture in a "bedroom." The CNN architecture, such as ResNet, VGG, and EfficientNet, extracts hierarchical features through multi-layer convolution, including edges, textures, object parts, and the overall scene. The image encoder extracts image features, and then uses backbone network optimization and data augmentation to achieve environmental feature focus. Backbone network optimization is targeted at environmental classification tasks, training the network to focus on scene-relevant features, such as the bed and windows in a bedroom, or the tree trunks and shrubs in a forest. Data enhancement uses rotation, scaling, and color adjustment to simulate real-world scene lighting changes and improve robustness.

[0035] In step S13, the image feature vector and the image text description are aligned through a multimodal alignment and fusion method, and the cross-attention mechanism and graph neural network are used to perform feature fusion to generate fused multimodal features. Specifically, the image feature vector and the embedding vector of the text token are unified to the same dimension through the projection layer; then, through comparative learning, InfoNCE Loss is used to make the first language model learn that the image features corresponding to a certain environmental category are similar to the embedding of environmental feature objects, while maximizing the difference in embedding with other environmental categories. In addition, a context fusion mechanism is adopted, and the cross-attention mechanism allows attention to key areas in the image, such as focusing on the "bed" when the environmental category is identified as "bedroom". At the same time, a graph neural network (GNN) is used to construct a relationship diagram between the image and the region, such as occlusion between objects and spatial layout.

[0036] In step S14, the multimodal features are input into a classifier, the environment type is output, and the environment description is generated in combination with the image text description.

[0037] Specifically, the classifier includes a fully connected layer and a label alignment layer. The fully connected layer inputs the fused multimodal features into the classifier and outputs probabilities for environment categories, such as bedroom, lawn, and beach. The label alignment layer uses the label data to establish a mapping between image features and image text labels. Step S14 ultimately outputs the environment type and description, for example, "The drone array is located in a bedroom, furnished with a bed, wardrobe, and other furniture, with an area of 15 square meters, a height of 2.8 meters, and a limited range of motion." "Bedroom" is the environment type, "bed" and "wardrobe" have corresponding image text labels in the first language model. The area and height can be measured in real time using the drone's built-in sensors. "Limited range of motion" is the descriptive text output by the first language model through inference. The first language model's database contains a large number of mappings between environment types and image labels, as well as mappings between environment types and corresponding range restrictions. The first language model was trained using hundreds of images of different environment types, each containing several characteristic objects that reflect the environment type. Therefore, the trained first language model can identify and describe a variety of environmental types. Therefore, the method of controlling a drone swarm formation based on a large language model in this application can control a drone swarm or array to improve its comprehensive understanding and comprehension of the surrounding environment, providing a strong guarantee for the real-time control of drone swarms in complex scenarios. Steps S11-S14 can be understood as the pre-training process of the first language model. By recognizing a large number of images of known environmental types, the recognition accuracy and precision of the first language model are continuously optimized.

[0038] After identifying the environment type, this application further extracts and analyzes the user instructions to generate precise command semantics. Figure 3 As shown, step S2, obtaining user instructions and environmental data, parsing the user instructions and the environmental data based on the pre-trained second language model, and generating command semantics in combination with the environmental description, includes the following steps: S21, receiving the user instruction and the environmental data collected by the sensor through the pre-trained second language model, parsing and extracting a command to be executed, where the user instruction includes a text instruction and a voice instruction; S22, based on the environment description, determining the risk of the command to be executed using the second largest language model; S23, extracting valid commands, sorting the valid commands, and generating the command semantics.

[0039] Specifically, the second language model is based on the ERNIE Speed language model developed by Baidu or a language model that can judge the risk of user instructions and accurately extract effective risk-free instructions. In this application, the second language model based on the ERNIE Speed language model is first pre-trained, such as Figure 4 As shown, the pre-training steps include: S2-1, constructing a question-answering dataset and performing data preprocessing, wherein the question-answering dataset includes 20,000 dialogue instances involving simultaneous coordination commands of multiple drones; S2-2, uses supervised fine-tuning methods to adjust the second language model, change model parameters, adjust model architecture and training strategy; S2-3, integrate the visual editor and command encoder into the second largest language model, optimize the output layer architecture, and use mixed precision and distributed training methods for the pre-training.

[0040] Specifically, the question-answering dataset includes 20,000 examples of conversations involving synchronized, coordinated commands for multiple drones. For example, sentences like "The drone swarm is currently in a factory with a medium operating range. The first drone is avoiding the obstacle ahead. The second drone is avoiding danger below it. The third drone is ascending significantly." Also, sentences like "The drone swarm is currently in a bedroom with a small operating range. The first drone is retreating half a meter. The second drone is ascending 100 centimeters. The third drone is retreating slightly." These examples involve synchronized, coordinated commands for multiple drones. This dataset was used to train a pre-trained large-scale language model specifically designed to understand and parse human intent. The model is capable of understanding direct or ambiguous user instructions and extracting precise control commands from them.

[0041] Specifically, during pre-training, the second language model is fed with user commands, environmental data collected by drone sensors, and the environmental description output by the first language model. User commands can be text or voice commands, such as "First drone, move forward a little" or "Form a triangle." Environmental data collected by drone sensors includes each drone's latitude and longitude, altitude, relative height within the environment, ambient temperature, weather, and other environmental information.

[0042] Step S2-2 uses a supervised fine-tuning method to adjust the second language model, changing model parameters, adjusting the model architecture and training strategy, specifically including: Data preprocessing: converting natural language instructions into specific control parameters, such as flight paths and cluster coordination instructions; Model adaptation: Adjust the architecture or training strategy of the lightweight original ERNIE Speed model through methods such as knowledge distillation and mixed precision training to adapt to the real-time and low-latency requirements of drone control.

[0043] During data preprocessing, text enhancement and scene enhancement are implemented to further improve the accuracy of extracted instructions. For example, synonym replacement can be used to change "maintain a 5-meter distance" to "maintain a safe distance of 5 meters." In another embodiment, weather simulation and dynamic obstacle injection are implemented to enhance environmental risk assessment capabilities.

[0044] Specifically, the knowledge distillation design includes a teacher model and a student model. The teacher model is pre-trained based on ERNIE 3.0-Large (175B parameters), and its outputs include text comprehension (command intent classification, such as formation / obstacle avoidance) and visual reasoning, such as scene understanding features. The student model is based on ERNIE Speed-Drone (1.1B parameters), and the distillation strategy includes feature matching and corresponding alignment.

[0045] Specifically, mixed precision training includes precision configuration and parallel strategy. The precision configuration includes setting the learning rate to 3e-5 and the iteration to 4. After about 3 hours of training, the training loss is about 0.124.

[0046] Step S2-3, integrating the visual editor and the command encoder into the second language model, optimizing the output layer architecture, and performing the pre-training using a mixed precision and distributed training method, including: Multimodal processing: Combine the environment description output by the first language model, integrate the visual editor and command encoder for multimodal training; Evaluation and optimization: Design evaluation indicators suitable for drone control tasks, such as command accuracy and response time.

[0047] Specifically, the visual editor adopts lightweight ViT, utilizes ViT-Hybrid architecture and LiDAR point cloud processing to realize multimodal extraction of three-dimensional obstacle features, and splices them with visual features to achieve multimodal fusion.

[0048] ViT's lightweight design reduces computational complexity, improves real-time control, and is adaptable to drone scenarios, compatible with the image encoder of the leading language model. The Hybrid encoder uses a lightweight CNN with 3×3 convolution (stride 2) and pooling to extract local features and employs multi-head self-attention for layer-normalized fusion. LiDAR point cloud processing takes a 3D point cloud as input, performs domain sampling at the local feature layer, constructs a local connection graph, aggregates the global topology at the global feature layer, restores the point cloud resolution through upsampling, and enhances details. The final output is a 256-dimensional dense feature vector that expresses the overall semantics.

[0049] During the splicing process with visual features, the environment description output by the first language model is combined with the overall semantics of the second language model and fused with the instructions output by the instruction encoder for feature alignment and dimension splicing, enabling accurate and effective instructions to be extracted from ambiguous user instructions. In actual drone swarm control, user instructions may be ambiguous. Through the above pre-training process, the second language model can extract accurate control instruction sequences from ambiguous user instructions. However, control instruction sequences may contain risky situations. For example, if the environment description output by S1 is "The drone swarm is currently in a bedroom with a small operating range," and the user instruction is "Move forward 60 meters as a whole," the pre-trained large language model will analyze the user instruction and use the drone sensors to obtain the length, width, and height of the bedroom, as well as the distance between each drone and the bedroom wall. Through inference, it will derive the risk-free instructions of "move as a whole" and "move forward synchronously," as well as the risky instruction of "move 60 meters." For risky instructions, the second language model outputs risk warning information to the computer.

[0050] Specifically, if Figure 5 As shown, step S3, obtaining a preset formation image, calling the first language model to parse the preset formation image, generating a formation instruction in combination with the command semantics, and controlling the UAV array to complete the cluster formation according to the formation instruction, includes the following steps: S31, acquiring the preset formation image; S32, calling the first language model to parse the preset formation image and extract preset formation coordinates; S33, fusing the preset formation coordinates and the command semantics to generate a formation instruction and sending it to the pilot UAV; S34, the pilot drone controls the drone array to complete the cluster formation and feeds back the actual formation coordinates of any drone.

[0051] The preset formation image can be sent from a computer or captured by the pilot drone's camera. For example, a user displays the preset formation image to the pilot drone, which then captures the displayed image in real time and sends it to the first language model. The first language model parses the preset formation image and extracts the formation instructions, including the preset formation coordinates. The preset formation coordinates are then combined with the command semantics output by the second language model to generate a valid formation instruction that is sent to the pilot drone, which then sends it to the other follower drones. This process requires inferring the range of the preset formation coordinates, which must not exceed a safe range.

[0052] Optionally, the preset formation image can be a triangle, rectangle, pentagon, or irregular shape. The formation is determined from the outside inwards based on the number of drones in the array. For example, if a drone array contains four drones and a triangle formation is required, the lead drone and two follower drones are prioritized at the three vertices of the triangle, with the fourth drone positioned within the triangle.

[0053] Specifically, the formation strategy of this application adopts the leader-follower method. According to the formation specification, the required geometric formation shape can be obtained by the absolute position of the UAV. , relative displacement ,distance ,position and interior angles In this application, considering that the positioning system in the sensor can provide the absolute position of any UAV, the absolute position is used to describe the formation position of the UAVs. Therefore, the formation control law of each UAV can be designed as formula (1).

[0054] , (1) in, It's a drone The three-dimensional moving speed is also the control instruction given to the drone. V represents the index set containing all drones, is the gain parameter used to adjust the convergence speed, is the real-time position of the drone obtained from the positioning map, is the expected position decomposed from the expected formation and obtained from the output of the large language model.

[0055] Specifically, if Figure 7The figure shows the logical architecture of a control method for a drone swarm formation based on a multimodal large language model according to an embodiment of the present application. The method obtains an environmental image captured or received by a drone array, parses the environmental image using a first large language model to generate an environmental description. Environmental data is acquired through drone sensors, and user commands are obtained from the user. The acquired user commands, environmental data, and environmental description are input into a second large language model, which parses the user commands and environmental data. Command semantics are generated based on the environmental description, including determining the command and type. A preset formation image is then acquired and parsed using the first large language model. Coordinates are extracted from specific commands and transmitted to the second large language model. The second large language model combines the command semantics to generate formation commands, image understanding commands, and sensor call commands. Based on the formation commands, the drone array is controlled to complete the swarm formation. Image understanding commands are used by the first large language model to output image understanding information, and sensors are enabled to output sensor information. During the formation process, the final response and feedback are divided into three categories: action responses, image feedback, and sensor data feedback are generated for the drone array for each of the three commands. The feedback information or data is returned to the command and dispatch center via the pilot drone.

[0056] Specifically, if Figure 6 As shown, the second aspect of the present application is to provide a control system 1 for a drone cluster formation based on a multimodal large language model, comprising: The environment perception module 10 is configured to obtain an environment image captured or received by the drone array, and to parse the environment image using the first language model to generate an environment description; A command extraction module 20 is configured to obtain user instructions and environment data, parse the user instructions and the data based on a pre-trained second language model, and generate command semantics in combination with the environment description; a response generation module 30 configured to obtain a preset formation image, invoke the first language model to parse the preset formation image, generate a formation instruction based on the command semantics, and control the UAV array to complete the cluster formation according to the formation instruction; A database module 40, for storing question-answering datasets; The drone array 50 includes a pilot drone and several follower drones. The pilot drone is configured to capture or receive images of the environment and the predetermined formation, transmit formation instructions from the command and dispatch center to any of the follower drones, and provide feedback on the actual formation position of any of the follower drones to the command and dispatch center. The pilot drone and any of the follower drones are equipped with sensors for acquiring environmental data.

[0057] A third aspect of the present application provides a storage medium, which is a non-transitory computer-readable storage medium and stores a program for a method for controlling a swarm of drones based on a multimodal large language model. When the program is executed by a processor, the following steps are implemented: Obtaining an environment image captured or received by the drone array, and parsing the environment image using the first language model to generate an environment description; Obtain user instructions and environment data, parse the user instructions and the data based on the pre-trained second language model, and generate command semantics in combination with the environment description; Obtain a preset formation image, call the first language model to parse the preset formation image, generate a formation instruction based on the command semantics, and control the drone array to complete the cluster formation according to the formation instruction.

[0058] The following further describes a control method for a drone cluster formation based on a multimodal large language model of the present application through specific embodiments.

[0059] In one embodiment, a drone array is set up, consisting of a pilot drone numbered "1" and 20 follower drones, numbered 2-21. Both the pilot drone and the follower drones are equipped with sensors that collect information such as position and distance. The drone formation control method is tested in a factory and a lawn environment. Thirty user commands are provided, such as "The current image in front of the drone shows an object approximately 100 cm away. Based on this information, please describe the type of environment the drone is in and its range of movement to extract environmental information."

[0060] Drones collect environmental information through onboard sensors and cameras, then interpret this information using the GPT-4 image understanding model. Specifically, the GPT-4 model retrieves data such as the distance between the lead drone and obstacles ahead, camera images, and the specific position and altitude of each drone. Before extracting commands, the formation system feeds the GPT-4 model with real-time drone images and pre-set language prompts for environmental analysis. For example, a language prompt might include, "The current image in front of the drone shows an object approximately 100 centimeters away. Based on this information, please describe the type of environment the drone is in and its range of movement to extract environmental information." Based on these specific language prompts, the multimodal large language model GPT-4 returns a structured description of the environment and an estimated range of movement, such as, "The drone is currently in a bedroom, and its range of movement is small." The pre-trained large language model then integrates this environmental information with user commands. Based on this environmental information, it assesses the safety of the current user command, rejecting unsafe commands. Based on different user commands, sensor feedback, and environmental information, the model reassesses the risk level of the command, resulting in different command outputs. Multimodal GPT-4 and the pre-trained ERNIE Speed large language model play a decisive role in the command extraction module.

[0061] The pre-trained large language model takes as input the relevant commands issued by the user and information about the drone's surrounding environment. The output is the precise command extracted by the large language model, such as "(1, forward, 50)", where "1" represents the drone's serial number, "forward" is the command executed by the drone, and "50" is the distance. By leveraging the pre-trained large language model to accurately extract and parse natural language commands, the ultimate goal is to achieve real-time, efficient, and end-to-end control of drone swarms in several different environments.

[0062] In this embodiment, a self-instruction method was used to systematically design and generate approximately 600 control command sequences for drone groups, including environmental descriptions and text instructions that simulate user input. This is a method of inputting specific format prompts into GPT-4, allowing GPT-4 to batch generate multiple data sets. During the data generation process, the data generation similarity can be set to filter and expand the data set. These instructions are used with the pre-trained ERNIE Speed large language model to evaluate its ability to understand and accurately execute complex formation control instructions. These data are also divided into four categories: safe instruction sets, dangerous instruction sets, mixed instruction sets, and non-existent instruction sets. By statistically analyzing the parsing and response results of each instruction, this application calculated that the average accuracy of the large language model in processing such high-level and coordinated tasks reached 82.7%. The calculation method is shown in Formula (2), and the specific statistical data are shown in Table 1.

[0063] , (2) Table 1

[0064] To test the multimodal GPT-4's image processing capabilities, this application selected dozens of examples of regular geometric shapes and irregular graphics. These preset formation images were input one by one into the GPT-4 large language model to test the model's ability to accurately extract spatial coordinate information from image input (visual input). After a series of experimental operations, this application also quantitatively evaluated the model's performance in this image recognition and coordinate positioning task and calculated the corresponding accuracy, which reached 83.8%. The accuracy calculation method is shown in Formula (3).

[0065] , (3) Table 2

[0066] To ensure that formation control works well in drones and that the proposed drone swarm formation method, which combines two large language models, is used, we tested the flight accuracy of a 21-drone array in various environments, including a foreground and a lawn. The test results are shown in Table 2, where the small difference between the target position and the actual hovering position demonstrates the high accuracy of the drone flight.

[0067] In summary, the present invention's method for controlling drone swarm formations based on a multimodal large language model provides an interactive control method for drone swarms. This method uses a multimodal large language model to analyze image data captured by the drone swarm, deeply integrating user voice or text input, environmental recognition information, and the real-time status of the drone swarm. It accurately extracts appropriate control commands based on the user's natural language commands, enabling smooth interaction with human users. When receiving ambiguous or potentially dangerous human commands, the pre-trained large language model leverages common sense reasoning and dataset learning to accurately discern user intent and filter out valid operational commands, thereby rejecting potentially risky actions. The average command extraction accuracy reaches 82.7%. Specifically, the present invention uses a multimodal GPT-4 large language model to process images captured by a pilot drone group and perform swarm formation control based on image features. By obtaining the corresponding coordinates and changing the drone swarm's formation configuration, an accuracy rate of 83.8% is achieved, effectively advancing formation control technology based on visual features. This breakthrough not only demonstrates the enormous potential of large language models in multi-agent visual perception and collaboration but also provides key technical support for the efficient operation of large-scale drone swarms. In summary, this study proposed a practical and feasible application framework and training method, demonstrating the importance of large language models in enhancing the situational understanding ability, dynamic adaptability and mission execution efficiency of swarm UAV systems.

[0068] In summary, the present application provides a method for controlling a drone cluster formation based on a multimodal large language model, including: obtaining an environmental image taken or received by a drone array, parsing the environmental image through a first large language model to generate an environmental description; obtaining user instructions and environmental data, parsing the user instructions and the data based on a pre-trained second large language model, and generating command semantics in combination with the environmental description; obtaining a preset formation image, calling the first large language model to parse the preset formation image, generating formation instructions in combination with the command semantics, and controlling the drone array to complete the cluster formation according to the formation instructions. The present application uses a multimodal large language model and a supervised large language model to not only analyze the environment, but also accurately identify effective instructions, eliminate risky instructions, and identify image formation instructions, thereby providing key technical support for the efficient operation of large-scale drone groups and verifying the effectiveness of large language models in multi-agent visual perception and collaboration. The present application solves the problem of real-time drone cluster control based on multimodal image acquisition and recognition.

[0069] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A UAV swarm formation control method based on a multimodal large language model, characterized in that: The following steps are involved: Obtaining an environment image captured or received by the drone array, and parsing the environment image using the first language model to generate an environment description; Obtaining user instructions and environment data, parsing the user instructions and the environment data based on the second language model, and generating command semantics in combination with the environment description; Obtain a preset formation image, call the first language model to parse the preset formation image, generate a formation instruction based on the command semantics, and control the drone array to complete the cluster formation according to the formation instruction.

2. The method for controlling a swarm of drones based on a multimodal large language model according to claim 1, characterized in that: The drone array includes at least one pilot drone and several follower drones; the pilot drone is used to capture or receive the environmental image and the preset formation image; the pilot drone and any of the follower drones are equipped with several sensors for acquiring the environmental data.

3. The method for controlling a swarm of drones based on a multimodal large language model according to claim 1, wherein: The first large language model is the multimodal GPT-4 large language model.

4. The method for controlling a swarm of drones based on a multimodal large language model according to claim 2, wherein: The step of obtaining an environment image captured or received by an array of drones and parsing the environment image using a first language model to generate an environment description includes the following steps: The pilot drone captures or receives the environment image and sends it to the first language model; The first language model uses an image encoder to extract image features and focus on environmental features to generate an image feature vector; The image feature vector and the image text description are aligned through a multimodal alignment and fusion method, and the cross-attention mechanism and graph neural network are used to perform feature fusion to generate fused multimodal features; The multimodal features are input into a classifier, the environment type is output, and the environment description is generated by combining the image text description.

5. The method for controlling a swarm of drones based on a multimodal large language model according to claim 2, wherein: The method of obtaining user instructions and environment data, parsing the user instructions and environment data based on the second language model, and generating command semantics in combination with the environment description includes the following steps: receiving the user instructions and the environmental data collected by the sensor through the second largest language model, parsing and extracting commands to be executed, wherein the user instructions include text instructions and voice instructions; Based on the environment description, determining the risk of the command to be executed by using the second largest language model; Valid commands are extracted, sorted, and the command semantics are generated.

6. The method for controlling a swarm of drones based on a multimodal large language model according to claim 1, wherein: The second largest language model is the ERNIE Speed large language model. The pre-training process of the second largest language model includes: Constructing a question-answering dataset and performing data preprocessing, wherein the question-answering dataset includes several dialogue instances involving synchronized coordinated commands of multiple drones; Use supervised fine-tuning methods to adjust the second language model, change model parameters, adjust model architecture and training strategy; The visual editor and command encoder are integrated into the second language model, the output layer architecture is optimized, and the pre-training is performed using mixed precision and distributed training methods.

7. The method for controlling a swarm of drones based on a multimodal large language model according to claim 6, characterized in that: The visual editor adopts lightweight ViT, utilizes ViT-Hybrid architecture and LiDAR point cloud processing to realize multimodal extraction of three-dimensional obstacle features, and splices them with visual features to achieve multimodal fusion.

8. The method for controlling a swarm of drones based on a multimodal large language model according to claim 2, wherein: The method of obtaining a preset formation image, calling the first language model to parse the preset formation image, generating a formation instruction based on the command semantics, and controlling the UAV array to complete the cluster formation according to the formation instruction includes the following steps: Acquiring the preset formation image; calling the first language model to parse the preset formation image and extract the preset formation coordinates; Fusion of the preset formation coordinates and the command semantics to generate a formation instruction and send it to the pilot drone; The pilot drone controls the drone array to complete the cluster formation and feeds back the actual formation coordinates of any drone.

9. A UAV swarm formation control system based on a multimodal large language model, characterized by: include: An environment perception module is used to obtain an environment image captured or received by the drone array, and parse the environment image using the first language model to generate an environment description; A command extraction module is used to obtain user instructions and environment data, parse the user instructions and the environment data based on the second language model, and generate command semantics in combination with the environment description; The response generation module is used to obtain a preset formation image, call the first language model to parse the preset formation image, generate a formation instruction based on the command semantics, and control the drone array to complete the cluster formation according to the formation instruction.

10. A storage medium, characterized in that: The storage medium is a non-transitory computer-readable storage medium, storing a program of a method for controlling a swarm of drones based on a multimodal large language model according to any one of claims 1 to 8, and when the program is executed by a processor, the following steps are implemented: Obtaining an environment image captured or received by the drone array, and parsing the environment image using the first language model to generate an environment description; Obtaining user instructions and environment data, parsing the user instructions and the environment data based on the second language model, and generating command semantics in combination with the environment description; Obtain a preset formation image, call the first language model to parse the preset formation image, generate a formation instruction based on the command semantics, and control the drone array to complete the cluster formation according to the formation instruction.

Images