Unmanned aerial vehicle group intelligent command and control system based on large model

Abstract

According to the unmanned aerial vehicle group intelligent command and control system based on the large model, the multi-modal input data is obtained through the data processing module, the multi-modal input data is processed to obtain the command data and the task data, the task plug-in is called according to the command data, and the task data is processed through the task plug-in to obtain the return data; the target information is displayed on the man-machine interaction module according to the returned data; receiving a task instruction generated according to the target information, and calling a task plug-in according to the task instruction to control the unmanned aerial vehicle group to execute the target task; the man-machine interaction module is used for receiving and displaying target information, the data processing module is used for understanding input instructions, integrating and calling plug-ins, integrating and outputting processing results and the like, and the task plug-ins are embedded into the data processing module and have specific task processing capacity. The intelligent interaction convenience of the large model and the professional ability of traditional task processing are combined, and the command control efficiency is high.

Description

Technical Field

[0001] The present application relates to the technical field of unmanned aerial vehicle mission control, and in particular to an intelligent command and control system for unmanned aerial vehicle swarms based on a large model. Background Art

[0002] In recent years, artificial intelligence technology has developed rapidly, and large models (such as GPT, BERT, etc.) have demonstrated powerful capabilities in natural language processing, image recognition, data analysis, etc. However, the current drone swarm control system has poor task execution flexibility, and a single large model is difficult to meet the actual needs of complex and diverse drone swarm command and control. Summary of the invention

[0003] In view of this, the embodiment of the present application proposes a drone swarm intelligent command and control system based on a large model, which aims to realize the embedding of task plug-ins based on the large model, combines the convenience of intelligent interaction of the large model with the professional capabilities of traditional task processing, and has high command and control efficiency.

[0004] To achieve the above-mentioned objectives, an embodiment of the present application provides a large-model-based intelligent command and control system for a drone swarm, comprising: a data processing module and a human-computer interaction module and an open integrated task plug-in that are communicatively connected to the data processing module, wherein the data processing module is constructed based on a fine-tuned large model as a base; wherein the data processing module is used to obtain multimodal input data, process the multimodal input data to obtain instruction data and task data, call the open integrated task plug-in according to the instruction data, and process the task data according to the open integrated task plug-in to obtain return data, and display target information in the human-computer interaction module according to the return data; receive a task instruction generated according to the target information, and call the open integrated task plug-in according to the task instruction to control the drone swarm to perform the target task; and the human-computer interaction module is used to receive and display the target information.

[0005] Optionally, the data processing module includes: an input unit, a multimodal data processing unit, a natural language processing engine unit, a semantic analysis unit and a task scheduling engine unit which are communicatively connected in sequence: wherein the input unit is used to respectively receive multimodal input data collected from multiple data sources, wherein the multimodal input data includes a first text data, voice data, image data and video data; the multimodal data processing unit is used to process the multimodal input data to obtain multimodal output data; the natural language processing engine unit is used to process the acquired instruction data to obtain intent recognition data; the semantic analysis unit is used to process the intent recognition data to obtain a plug-in call instruction; the task scheduling engine unit is used to call the corresponding plug-in in the open integrated task plug-in according to the plug-in intelligent recommendation algorithm and the plug-in call instruction to process the task data to obtain a multimodal task instruction.

[0006] Optionally, the natural language processing engine unit includes a natural language recognition subunit and a text preprocessing subunit that are communicatively connected: wherein the natural language recognition subunit is used to parse the input instruction data to obtain second text data; and the text preprocessing subunit is used to call the NPL library to process the second text data to obtain intent recognition data.

[0007] Optionally, the semantic analysis unit includes a semantic role labeling subunit and a semantic similarity calculation subunit that are communicatively connected; wherein the semantic role labeling subunit is used to identify the predicates and semantic roles in the intent recognition data to obtain the user intent instruction; and the semantic similarity calculation subunit is used to calculate the semantic similarity between the user intent instruction and the preset instruction to obtain the plug-in call instruction.

[0008] Optionally, the semantic role labeling subunit is constructed based on the SRL model in AllenNLP, and the semantic similarity calculation subunit is constructed based on the Word2Vec model, the GloVe model or the BERT model of Transformer.

[0009] Optionally, the task scheduling engine unit includes a message queue subunit, a task allocation subunit, a task execution subunit and a task return subunit: wherein the message queue subunit is used to receive the plug-in type and the occupancy status of each plug-in in the open integrated task plug-in according to the plug-in call instruction, and dynamically arrange the plug-ins based on the plug-in type and the occupancy status of each plug-in; the task allocation subunit is used to allocate pre-processed data to each dynamically arranged plug-in; the task execution subunit is used to call each plug-in to process the pre-processed data and obtain multi-modal task instructions; the task return subunit is used to obtain the multi-modal task instructions.

[0010] Optionally, the message queue subunit is constructed based on a RabbitMQ message queue system or a Kafka message queue system.

[0011] Optionally, the open integration task plug-in includes: a plug-in management unit, a plug-in directory unit, a plug-in loader and an intelligent recommendation unit that are communicatively connected: wherein the plug-in management unit is responsible for registering, loading and unloading each plug-in to the data processing module, wherein each plug-in has a plug-in code, a configuration file and metadata; the plug-in directory unit is used to maintain the directory structure of each plug-in; the plug-in loader is used to scan the directory structure, determine the metadata of the plug-in based on the directory structure, and initialize each plug-in based on the metadata and the configuration file; the intelligent recommendation unit is used to analyze the historical call data and user behavior of each plug-in according to the intelligent recommendation algorithm, and recommend each plug-in to the data processing module.

[0012] Optionally, the human-computer interaction module includes an API interface and a graphical user interface: wherein the API interface is used to input the multimodal input data into the data processing module and receive target information output by the data processing module, and send the target information to the graphical user interface; the graphical user interface is used to receive multimodal input data and send the multimodal input data to the API interface, as well as receive the target information and display the target information.

[0013] Optionally, the API interface adopts a RESTful architecture and performs data queries and operations based on GraphQL.

[0014] The present application proposes an intelligent command and control system for drone swarms based on a large model, which is constructed based on a fine-tuned large model through a data processing module, a human-computer interaction module and an open integrated task plug-in that are communicatively connected to the data processing module; wherein the data processing module is used to obtain multimodal input data, wherein the multimodal input data includes instruction data and task data, the data processing module identifies the instruction data to obtain a plug-in call instruction, calls the open integrated task plug-in according to the plug-in call instruction, and processes the task data according to the open integrated task plug-in to obtain return data, and displays target information in the human-computer interaction module according to the return data; receives task instructions generated according to the target information, and calls the open integrated task plug-in according to the task instruction to control the drone swarm to perform the target task; the human-computer interaction module is used to receive and display the target information, the data processing module is used to understand the input instructions, integrate and call the plug-in, integrate and output the processing results, etc., the open integrated task plug-in is embedded in the data processing module, and has a specific task processing capability. The present invention implements the embedding of task plug-ins based on large models, combines the convenience of intelligent interaction of large models with the professional capabilities of traditional task processing, and has high command and control efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a general structural block diagram of a large-model-based UAV swarm intelligent command and control system provided in one embodiment of the present application;

[0016] Figure 2 It is a schematic diagram of data access and multi-modal processing of a large-model-based drone swarm intelligent command and control system provided in one embodiment of the present application;

[0017] Figure 3 It is a structural block diagram of a data processing module of a large-model-based drone swarm intelligent command and control system provided in one embodiment of the present application;

[0018] Figure 4 It is a structural block diagram of an open integrated task plug-in of a large model-based UAV swarm intelligent command and control system provided in one embodiment of the present application;

[0019] Figure 5 It is a structural block diagram of the human-computer interaction module of the large-model-based drone swarm intelligent command and control system provided in another embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0021] The UAV swarm intelligent command and control system proposed in this application is based on a large model. Figure 1The command and control system 100 may include: a data processing module 101 and a human-computer interaction module 102 and an open integrated task plug-in 103 that are communicatively connected to the data processing module 101, wherein the data processing module 101 is constructed based on a fine-tuned large model as a base; wherein the data processing module 101 is used to obtain multimodal input data, process the multimodal input data to obtain instruction data and task data, call the open integrated task plug-in 103 according to the instruction data, and process the task data according to the open integrated task plug-in 103 to obtain return data, and display target information in the human-computer interaction module 102 according to the return data; receive a task instruction generated according to the target information, and call the open integrated task plug-in 103 according to the task instruction to control the drone group to perform the target task; the human-computer interaction module 102 is used to receive and display the target information.

[0022] Specifically, the command and control system uses a large model fine-tuned with industry knowledge as the base, performs intelligent processing of multimodal data, realizes language understanding of drone command and control instructions, open integration and intelligent calling of task plug-ins, and supports drone group target recognition, voice recognition, task decision-making, etc. The system consists of a data processing module 101, an open integrated task plug-in 103 and a human-computer interaction module 102. The data processing module 101 is used to understand input instructions, call the open integrated task plug-in 103, and integrate and output processing results.

[0023] It should be noted that the data processing module 101 is centered on the large model base, which carries the training and reasoning tasks of the generative large model. Through massive data pre-training and fine-tuning of specific tasks, the large model base can realize intelligent processing of multimodal data.

[0024] The open integrated task plug-in 103 is embedded in the data processing module through a specific interface. The open integrated task plug-in 103 has specific task processing capabilities and is used for data processing in specific task scenarios, such as image target recognition, speech recognition, resource scheduling, etc.

[0025] It should be particularly pointed out that the open integrated task plug-in 103 in the present invention allows third-party developers to integrate their own task plug-ins into the present control system to achieve flexible expansion and customization of functions.

[0026] The present invention implements the embedding of the open integrated task plug-in 103 based on the big model, and combines the big model preset in the data processing module 101 to realize the convenience of interaction and the professionalism of task processing, so that the command and control system has a higher command and control efficiency.

[0027] See also Figure 3In an embodiment of the present application, the data processing module 101 includes: an input unit 1011, a multimodal data processing unit 1012, a natural language processing engine unit 1013, a semantic analysis unit 1014 and a task scheduling engine unit 1015 which are sequentially connected in communication: wherein the input unit 1011 is used to respectively receive multimodal input data collected from multiple data sources, wherein the multimodal input data includes first text data, voice data, image data and video data; the multimodal data processing unit 1012 is used to process the multimodal input data to obtain multimodal output data; the natural language processing engine unit 1013 is used to process the acquired instruction data to obtain intent recognition data; the semantic analysis unit 1014 is used to process the intent recognition data to obtain a plug-in call instruction; the task scheduling engine unit 1015 is used to call the corresponding plug-in in the open integration task plug-in 103 according to the plug-in intelligent recommendation algorithm and the plug-in call instruction to process the task data to obtain a multimodal task instruction.

[0028] In the actual application scenario, the data processing module 101 is built based on the intelligent command and control framework of the drone swarm of the large model. The data processing module 101 first receives multimodal data from various sources through its data access layer, where the multimodal data can be text, image, audio and video. It should be noted that the multimodal data interface is an important part of the data processing module 101 and is also the input unit 1011 of the data processing module 101. The input unit 1011 supports obtaining data from multiple data sources such as text, image, audio and video.

[0029] It should be noted that data collection is a step before data input, and the input unit 1011 is responsible for obtaining raw data from various data sources, and the raw data mainly includes the following data types: text data, image data, audio data, and video data. The collected raw data can be input into the data processing module 101 through the input unit 1011, wherein the raw data can include instruction data and task data, for example, audio data and text data can be instruction data, and video data and image data can be task data.

[0030] In another embodiment, the input unit 1011 may also automatically obtain original data from the data source.

[0031] like Figure 2As shown, data preprocessing is a key step to ensure data quality, including operations such as data cleaning, data labeling and data normalization, which can be completed by the multimodal data processing unit 1012. Data cleaning includes removing duplicate data and processing missing values. Data labeling is divided into manual labeling and automatic labeling. Manual labeling is to use labeling tools (such as LabelImg) for manual labeling. The labeling is accurate but inefficient. It is usually used in the initial stage of self-supervised training to create a small-scale precision label data set and obtain a pre-trained model under strong supervision. Automatic labeling is to use a pre-trained model for self-supervised labeling. Commonly used technologies include NLP libraries (such as spaCy and NLTK). The multimodal data processing unit 1012 can continuously iterate the model based on the inference results of the pre-trained model on the unlabeled image as weakly supervised data. Data normalization is used to process different types of data to make it conform to the input form of the model. For different types of data, such as numerical normalization using StandardScaler or MinMaxScaler of the scikit-learn library, image normalization using PIL or OpenCV library, and audio normalization using librosa library.

[0032] In the specific implementation process, first, after completing data preprocessing, the multimodal data processing unit 1012 can perform multimodal processing on the preprocessed data to obtain a plug-in calling instruction, and the plug-in calling instruction can call the corresponding plug-in to process the preprocessed data during the multimodal data processing.

[0033] It should be noted that the plug-in calling instruction can be obtained based on voice data or text data. If it is obtained based on voice data, the multimodal data processing unit 1012 needs to call the natural language processing engine unit 1013 and the semantic analysis unit 1014 to complete the processing of the voice data and obtain the plug-in calling instruction.

[0034] In an embodiment of the present application, the natural language processing engine unit 1013 includes a natural language recognition subunit and a text preprocessing subunit that are communicatively connected: wherein the natural language recognition subunit is used to parse the input instruction data to obtain second text data; and the text preprocessing subunit is used to call the NPL library to process the second text data to obtain intent recognition data.

[0035] Among them, first, after completing the data preprocessing and completing the multimodal processing of the input data, a large amount of data needs to be processed and understood. The natural language processing engine unit 1013 uses advanced natural language processing (NLP) technology to deeply analyze and semantically understand the user's input instructions. The natural language processing engine unit 1013 can accurately understand the user's intentions and needs through technologies such as word segmentation, part-of-speech tagging, and syntactic analysis. The natural language processing engine unit 1013 converts natural language instructions into machine-understandable intention recognition data in a text format. This accurate understanding of input instructions is the key to the platform's intelligent processing.

[0036] In the specific implementation process, after the natural language processing engine unit 1013 preliminarily understands the user's intention, it can use the semantic analysis unit 1014 to perform word segmentation, part-of-speech tagging, named entity recognition (NER) and other processing on the intention recognition data in text format. Specifically, the semantic analysis unit 1014 can use a classification model or a rule system to identify the user's intention, parse the grammatical structure of the sentence, and understand the relationship between words. The semantic analysis unit 1014 is used to pre-process the text using an NLP library (such as spaCy, NLTK) in text processing, including word segmentation, part-of-speech tagging and named entity recognition. Then, the semantic analysis unit 1014 further analyzes the semantic information in the instruction to ensure that the command system can accurately execute the user request. The semantic analysis unit 1014 can identify the predicate and its semantic role in the sentence through semantic role labeling (SRL), such as the agent, the patient, etc. to identify the object referred to in the text and understand the contextual relationship. The semantic analysis unit 1014 can also calculate the semantic similarity between the user instruction and the possible system response.

[0037] In an embodiment of the present application, the semantic analysis unit 1014 may include a semantic role labeling subunit and a semantic similarity calculation subunit that are communicatively connected; wherein the semantic role labeling subunit is used to identify the predicates and semantic roles in the intent recognition data to obtain the user intent instruction; and the semantic similarity calculation subunit is used to calculate the semantic similarity between the user intent instruction and the preset instruction to obtain the plug-in call instruction.

[0038] Specifically, the semantic role labeling subunit is constructed based on the SRL model in AllenNLP, and the semantic similarity calculation subunit is constructed based on the Word2Vec model, the GloVe model or the BERT model of Transformer.

[0039] In the specific implementation process, the semantic role labeling subunit uses a pre-trained semantic role labeling model (such as the SRL model in AllenNLP) to label, extract semantic roles, and use a coreference resolution model (such as NeuralCoref) to process reference relationships to ensure the coherence of instruction understanding. The semantic similarity calculation subunit can use a word embedding model (such as Word2Vec, GloVe, or Transformer-based BERT) to calculate the semantic similarity between instructions and system responses to ensure the accuracy of responses.

[0040] Further, after obtaining the plug-in calling instruction, the multimodal data processing unit 1012 calls the task scheduling engine unit 1015 according to the plug-in calling instruction, and calls the corresponding plug-in in the open integrated task plug-in 103 according to the plug-in intelligent recommendation algorithm and the plug-in calling instruction to process the task data to obtain a multimodal task instruction.

[0041] In an embodiment of the present application, the task scheduling engine unit 1015 includes: a communicatively connected message queue subunit, a task allocation subunit, a task execution subunit and a task return subunit: wherein the message queue subunit is used to receive the plug-in type and the occupancy status of each plug-in in the open integrated task plug-in 103 according to the plug-in call instruction, and dynamically arrange the plug-ins based on the plug-in type and the occupancy status of each plug-in; the task allocation subunit is used to allocate pre-processed data to each dynamically arranged plug-in; the task execution subunit is used to call each plug-in to process the pre-processed data and obtain multi-modal task instructions; the task return subunit is used to obtain the multi-modal task instructions.

[0042] The task scheduling engine unit 1015 automatically schedules the corresponding task plug-in in the open integrated task plug-in 103 for processing according to the plug-in call instruction, ensuring that the system efficiently responds to the plug-in call instruction. The task scheduling engine unit 1015 uses the message queue subunit to manage and schedule task requests, and dynamically allocates tasks to appropriate plug-ins for processing according to the task type and system resource conditions.

[0043] Furthermore, the message queue subunit is used to receive the plug-in type and occupancy of each plug-in in the open integrated task plug-in 103 according to the plug-in call instruction, and the plug-ins dynamically arranged based on the plug-in type and the occupancy of each plug-in can use the intelligent recommendation algorithm to dynamically arrange each task. Among them, the intelligent recommendation algorithm can be used by the large model base to recommend suitable task plug-ins and parameter settings according to the needs and historical data and user behavior, so as to improve the intelligence of the system and user experience. Based on user behavior data, task plug-ins and parameter settings that are liked by similar users are recommended. Based on the characteristics of the task plug-in and the needs of the user, the most suitable task plug-in is recommended. By continuously learning from user feedback, the recommendation algorithm is optimized to provide more accurate recommendations.

[0044] In an embodiment of the present application, the message queue subunit is constructed based on a RabbitMQ message queue system or a Kafka message queue system.

[0045] It should be particularly noted that the message queue subunit supports multi-task parallel processing, which improves the response speed and throughput of the system.

[0046] Please refer to Figure 4 In an embodiment of the present application, the open integration task plug-in 103 includes: a plug-in management unit 1031, a plug-in directory unit 1032, a plug-in loader 1033 and an intelligent recommendation unit 1034 connected in communication: wherein the plug-in management unit 1031 is responsible for registering, loading and unloading each plug-in to the data processing module 101, wherein each plug-in has a plug-in code, a configuration file and metadata; the plug-in directory unit 1032 is used to maintain the directory structure of each plug-in; the plug-in loader 1033 is used to scan the directory structure, determine the metadata of the plug-in according to the directory structure, and initialize each plug-in according to the metadata and the configuration file; the intelligent recommendation unit 1034 is used to analyze the historical call data and user behavior of each plug-in according to the intelligent recommendation algorithm, and recommend each plug-in to the data processing module 101.

[0047] In the specific implementation process, the task plug-in layer is the key part to achieve flexible function expansion. After completing the pre-training and fine-tuning of the data, the open integrated task plug-in 103 can provide a specific plug-in to call according to the plug-in call instruction, and support the access of third-party task plug-ins through the open plug-in architecture, so as to achieve rapid iteration and expansion of functions. The plug-in management unit 1031 is responsible for the registration, loading, unloading and version management of the plug-in, and the plug-in interface specification ensures the compatibility between different plug-ins.

[0048] The open integration task plug-in 103 allows third-party developers to develop and integrate plug-ins according to a unified interface specification, thereby enhancing the functionality and flexibility of the system. Each plug-in of the open integration task plug-in 103 is an independent service that can be independently developed, deployed and maintained. The plug-in loader 1033 dynamically discovers and loads plug-ins through configuration files or service registration centers. The plug-in directory unit 1032 defines a unified plug-in directory structure, and each plug-in contains its code, configuration files and metadata. The plug-in loader 1033 is also responsible for reading the plug-in directory, loading plug-ins and registering them into the system. The plug-in management unit 1031 uses message queues or local HTTP APIs to achieve communication and data exchange between plug-ins. The plug-in management unit 1031 is responsible for the full life cycle management of plug-ins, including registration, loading, unloading and version management. The plug-in management unit 1031 implements dynamic loading and management of plug-ins through dependency injection frameworks (such as Spring and Guice). The plug-in management unit 1031 uses a version control system (such as Git) to manage plug-in versions and dependencies. The plug-in management unit 1031 uses configuration management tools (such as Consul, Zookeeper) to manage the configuration and status of the plug-in. The plug-in interface specification defines the communication protocol and data format between the plug-in and the system and between the plug-ins to ensure the compatibility and scalability between different plug-ins. In the specific implementation process, the intelligent recommendation unit 1034 can intelligently call the corresponding module according to the plug-in call instruction, take the appropriate task plug-in for processing, and recommend the appropriate task plug-in and parameter settings according to the historical data and user behavior through the intelligent recommendation algorithm.

[0049] Please refer to Figure 5 In an embodiment of the present application, the human-computer interaction module 102 includes: an API interface 1021 and a graphical user interface 1022, wherein the API interface 1021 is used to input the multimodal input data to the data processing module 101 and receive the target information output by the data processing module 101, and send the target information to the graphical user interface 1022; the graphical user interface 1022 is used to receive the multimodal input data and send the multimodal input data to the API interface 1021, as well as receive the target information and display the target information.

[0050] Specifically, the API interface 1021 adopts a RESTful architecture and performs data queries and operations based on GraphQL.

[0051] By intelligently calling task plug-ins, the system processes specific tasks and generates output results, which are ultimately displayed in an intuitive form in the user interface. The user interaction layer is the interface that directly interacts with users in the multimodal data processing system. It provides an intuitive graphical user interface 1022 (GUI) and API interface 1021, which supports users to easily operate the platform and integrate and interact with other systems. This report will detail the design and implementation of the graphical user interface 1022 and API interface 1021, ensuring compatibility with the Llama 3 model and supporting offline deployment and maintenance.

[0052] The graphical user interface 1022 (GUI) provides an intuitive user interface to facilitate user interaction with the platform. A good GUI design can significantly improve the user experience and system usability. Use modern front-end frameworks (such as React, Vue.js, Angular) to build responsive and interactive user interfaces. Design interface elements as independent reusable components to improve development efficiency and code maintainability. Use state management libraries (such as Redux, Vuex) to manage the global state of the application to ensure data consistency and synchronous updates of the interface. The API interface 1021 provides integration and interaction with other systems, supports data acquisition and manipulation, and realizes the openness and interoperability of the system. The API interface 1021 can be designed using the RESTful architectural style to ensure the simplicity and ease of use of the interface. The API interface 1021 uses GraphQL for complex queries and data operations, providing a flexible and efficient API calling method.

[0053] An example:

[0054] The Qwen2-VL large model is selected as the base large model. After that, the command and control business data of the drone group is collected and trained using supervised learning, unsupervised learning or reinforcement learning methods, and LORA is used to fine-tune the large model. Based on the fine-tuned large model, task plug-ins such as speech recognition, image recognition, target positioning, combat planning, report generation, voice broadcasting, and data transmission and reception are integrated.

[0055] In a specific UAV group command and control, the framework is used to command and control four UAVs to perform ground reconnaissance and strike missions. During the target search process of the four UAVs, the framework receives the position, reconnaissance image and other data of the UAVs in real time through the data transceiver plug-in. The commander issues the "identify the vehicle in the image" command through voice. After the intelligent command and control framework performs voice recognition and intent understanding, it calls the image recognition task plug-in to identify the reconnaissance image of the UAV. After detecting the vehicle target, it automatically calls the target positioning task plug-in to solve the target position. The intelligent command and control framework comprehensively processes the calculation results of the task plug-in, generates a task report, and calls the voice broadcast plug-in to broadcast "UAV 1 finds the vehicle target, the position is 112.41° longitude, 24.67° latitude, and 120 meters altitude. Should it be attacked?" The commander issues the "attack target" command. After the intelligent command and control framework performs voice recognition and intent understanding, it calls the combat planning task plug-in to generate a combat plan "UAVs 1 and 2 continue to track the target and provide target instructions. UAV 3 launches a laser-guided missile attack", and sends the combat plan to each UAV through the data transceiver task plug-in for execution.

[0056] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A large-scale model-based UAV swarm intelligent command and control system, characterized in that: include: A data processing module and a human-computer interaction module and an open integrated task plug-in in communication with the data processing module, wherein the data processing module is constructed based on the fine-tuned large model; The data processing module is used to obtain multimodal input data, process the multimodal input data to obtain instruction data and task data, call the open integrated task plug-in according to the instruction data, and use the open integrated task plug-in to process the task data to obtain return data, display target information in the human-computer interaction module according to the return data, and receive task instructions generated according to the target information, and call the open integrated task plug-in according to the task instruction to control the drone group to perform the target task; The human-computer interaction module is used to receive and display target information.

2. The large-model-based UAV swarm intelligent command and control system according to claim 1 is characterized in that: The data processing module comprises: The input unit, multimodal data processing unit, natural language processing engine unit, semantic analysis unit and task scheduling engine unit are connected in sequence: The input unit is used to receive the multimodal input data collected by multiple data sources respectively, and the multimodal input data includes first text data, voice data, image data and video data; The multimodal data processing unit is used to process the multimodal input data to obtain the instruction data and the task data; The natural language processing engine unit is used to process the acquired instruction data to obtain intent recognition data; The semantic analysis unit is used to process the intention recognition data to obtain a plug-in calling instruction; The task scheduling engine unit is used to call the corresponding plug-in in the open integrated task plug-in according to the plug-in intelligent recommendation algorithm and the plug-in calling instruction to process the task data to obtain the multi-modal task instruction.

3. The large-model-based UAV swarm intelligent command and control system according to claim 2 is characterized in that: The natural language processing engine unit includes a natural language recognition subunit and a text preprocessing subunit which are communicatively connected: Wherein, the natural language recognition subunit is used to parse the input instruction data to obtain second text data; The text preprocessing subunit is used to call the NPL library to process the second text data to obtain the intent recognition data.

4. The large-model-based UAV swarm intelligent command and control system according to claim 2 is characterized in that: The semantic analysis unit comprises: A semantic role labeling subunit and a semantic similarity calculation subunit for communication connection; The semantic role labeling subunit is used to identify the predicate and semantic role in the intention recognition data to obtain the user intention instruction; The semantic similarity calculation subunit is used to calculate the semantic similarity between the user intended instruction and the preset instruction to obtain the plug-in calling instruction.

5. The large-model-based UAV swarm intelligent command and control system according to claim 4 is characterized in that: The semantic role labeling subunit is constructed based on the SRL model in AllenNLP, and the semantic similarity calculation subunit is constructed based on the Word2Vec model, the GloVe model or the BERT model of Transformer.

6. The large-model-based UAV swarm intelligent command and control system according to claim 2 is characterized in that: The task scheduling engine unit comprises: The message queue subunit, task allocation subunit, task execution subunit and task return subunit of the communication connection: The message queue subunit is used to receive the plug-in type and the occupancy of each plug-in in the open integration task plug-in called according to the plug-in calling instruction, and dynamically arrange the plug-ins based on the plug-in type and the occupancy of each plug-in; The task allocation subunit is used to allocate pre-processed data to each dynamically arranged plug-in; The task execution subunit is used to call each plug-in to process the pre-processed data to obtain multi-modal task instructions; The task return subunit is used to obtain the multi-modal task instructions.

7. The large-model-based UAV swarm intelligent command and control system according to claim 6 is characterized in that: The message queue subunit is constructed based on the RabbitMQ message queue system or the Kafka message queue system.

8. The large-model-based UAV swarm intelligent command and control system according to claim 1 is characterized in that: The open integration task plug-in includes: Communication connected plug-in management unit, plug-in catalog unit, plug-in loader and intelligent recommendation unit: The plug-in management unit is responsible for registering, loading and unloading each plug-in to the data processing module, and each plug-in has a plug-in code, a configuration file and metadata; The plug-in directory unit is used to maintain the directory structure of each plug-in; The plug-in loader is used to scan the directory structure, determine the metadata of the plug-in according to the directory structure, and initialize each plug-in according to the metadata and the configuration file; The intelligent recommendation unit is used to analyze the historical call data and user behavior of each of the plug-ins according to the intelligent recommendation algorithm, and recommend each of the plug-ins to the data processing module.

9. The large-model-based UAV swarm intelligent command and control system according to claim 1 is characterized in that: The human-computer interaction module comprises: API interface and graphical user interface: Wherein, the API interface is used to input the multimodal input data to the data processing module and receive the target information output by the data processing module, and send the target information to the graphical user interface; The graphical user interface is used to receive the multimodal input data and send the multimodal input data to the API interface, and to receive the target information and display the target information.

10. The large model-based UAV swarm intelligent command and control system according to claim 9 is characterized in that: The API interface adopts a RESTful architecture and performs data queries and operations based on GraphQL.

Images