An unmanned aerial vehicle cooperation system and multi-machine cooperation control method based on embodied intelligence

Abstract

The application discloses a kind of unmanned aerial vehicle cooperation systems and multi-machine cooperation control method based on embodied intelligence, wherein the unmanned aerial vehicle cooperation system includes instruction acquisition unit, instruction coding unit, environment perception unit, environment coding unit, fusion coding unit, action decoding unit and adaptive control unit, according to these units, instruction information and unmanned aerial vehicle sensing information can be efficiently fused, complex multi-modal environmental information and natural language form complex cooperation command can be understood, and calibration of control command or environment is not required by additional system, thereby meeting the collaborative work requirements of unmanned aerial vehicle under complex environmental conditions, and expanding the unmanned aerial vehicle use scenario.The multi-machine cooperation control method allows multiple unmanned aerial vehicles to access the cooperation system through adaptive control algorithms to adapt to the needs of different aircraft models working together.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) control technology, specifically to a UAV collaborative system based on embodied intelligence and a multi-UAV collaborative control method. Background Technology

[0002] Against the backdrop of the rapid development of artificial intelligence technology, drones, as an important intelligent tool, have demonstrated enormous application potential in various fields. Currently, drone collaboration has become a research hotspot, but existing drone collaboration systems still have many shortcomings: for example, the motion state and behavior strategies of ordinary drone collaboration systems are relatively fixed, unable to dynamically adjust with changes in the environment and task requirements, and lacking adaptability; their environmental understanding ability is poor, relying on complex number algorithms to analyze environmental features, and unable to achieve end-to-end understanding of complex environmental spaces; collaborative work depends on positional communication between drones; and the executed commands are relatively fixed, unable to understand more efficient natural language commands.

[0003] Based on the technological development trend and market demand for drone collaboration, there is an urgent need to provide an advanced and efficient solution. Summary of the Invention

[0004] This application aims to address the functional limitations caused by existing drone collaboration systems by providing a drone collaboration system based on embodied intelligence and a multi-drone collaborative control method.

[0005] According to a first aspect of this application, this application provides a drone collaborative system based on embodied intelligence, comprising: an instruction acquisition unit for acquiring voice instructions and text instructions; an instruction encoding unit, signal-connected to the instruction acquisition unit, for preprocessing the voice instructions and text instructions, and obtaining first encoded information through instruction encoding; an environment perception unit for acquiring multi-dimensional environmental sensing data of the drone; an environment encoding unit, signal-connected to the environment perception unit, for processing the multi-dimensional environmental sensing data, and obtaining second encoded information through data encoding; a fusion encoding unit, signal-connected to the instruction encoding unit and the environment encoding unit, for fusing the first encoded information from the instruction encoding unit and the second encoded information from the environment encoding unit, and generating fused encoded information through fusion; an action decoding unit, signal-connected to the fusion encoding unit, for decoding the fused encoded information to generate action control instructions for the drone; and an adaptive control unit, signal-connected to the action decoding unit, for predicting the motion state of the drone based on the action control instructions to obtain a state vector, and performing motion control on the drone according to the action control instructions and the state vector.

[0006] Furthermore, the instruction acquisition unit includes a voice instruction module and a text instruction module, which are integrated into a terminal controller for the UAV. The voice instruction module is used to generate the voice instruction by recognizing speech, and the text instruction module is used to generate the text instruction by recognizing text.

[0007] Furthermore, the instruction encoding unit includes an instruction preprocessing module and a first LLM encoding module integrated on the terminal controller for the UAV; the instruction preprocessing module is used to convert the voice instruction and the text instruction into a word vector matrix, and the first LLM encoding module is used to encode the word vector matrix into instructions and output the first encoded information.

[0008] Furthermore, the environmental perception unit includes a camera module, a laser point cloud module, and an attitude sensing module; the camera module, the laser point cloud module, and the attitude sensing module are integrated into the UAV. The camera module is used to generate RGB image data by capturing images of the environment in which the UAV is located. The laser point cloud module is used to generate laser point cloud data by scanning the environment in which the UAV is located using a laser. The attitude sensing module is used to collect the altitude, tilt angle, speed, and position of the UAV itself to generate flight attitude data. The RGB image data, the laser point cloud data, and the flight attitude data are integrated to form the multi-dimensional environmental sensing data.

[0009] Furthermore, the environment coding unit includes an environment preprocessing module and a second LLM coding module integrated on the terminal controller for the UAV; the environment preprocessing module is used to convert the multivariate environment sensing data into a sensing data matrix, and the second LLM coding module is used to encode the sensing data matrix and output the second coding information; the second coding information has streaming characteristics and can retain the dynamic features between consecutive frames.

[0010] Furthermore, the fusion encoding unit is integrated into the terminal controller for the UAV, and the fusion encoded information generated by fusing the first encoded information and the second encoded information adopts hybrid encoding tokens; the action decoding unit includes a multi-layer decoding module and an encoding feedforward module integrated into the terminal controller for the UAV, the multi-layer decoding module is used to decode the fusion encoded information into action features, and the encoding feedforward module is used to convert the action features into executable action control commands.

[0011] Furthermore, the adaptive control unit includes an end-to-end control module and a state prediction module; the state prediction module and the end-to-end control module are integrated into the UAV, the state prediction module is used to predict the motion state of the UAV based on the motion control command to obtain a state vector, and the end-to-end control module performs motion control on the UAV based on the UAV's flight attitude data and the state vector.

[0012] Furthermore, the drone is considered an intelligent agent, and the terminal controller for the drone is considered a decision-making core. The intelligent agent is able to accept the environmental influences of its working environment and adapt to the working environment through decision-making actions. The decision-making core is able to communicate with the intelligent agent, receive collected information from the intelligent agent and work instructions from the user, and push action control instructions to the intelligent agent.

[0013] According to a second aspect of this application, a multi-drone cooperative control method based on embodied intelligence is provided, comprising: checking the communication connection status of all drones and the terminal controller for the drones, and recording each successfully connected drone to obtain the number of vehicles; when the number of vehicles changes, adjusting the tensor size of the received input information and buffered output data according to the change in the number of vehicles; acquiring the flight status information and sensor status information of each successfully connected drone, and updating the crew state vector according to the flight status information and sensor status information of each drone; acquiring corresponding multi-dimensional environmental sensor data from drones in normal status based on the crew state vector; acquiring voice commands and text commands; and processing the... Voice commands and text commands are encoded to obtain first encoded information, and multi-source environmental sensor data is encoded to obtain second encoded information; the first encoded information and the second encoded information are fused to generate fused encoded information; the fused encoded information is decoded to generate motion control commands for one or more UAVs; the corresponding motion output tensors are calculated according to the motion control commands for one or more UAVs, and it is determined whether each motion output tensor exceeds a preset threshold; when a motion output tensor exceeds the threshold, the corresponding motion control command is set to zero, otherwise the corresponding motion control command is sent to the corresponding UAV.

[0014] The beneficial effects of this application are:

[0015] The aforementioned embodied intelligence-based UAV collaborative system and multi-UAV collaborative control method include a UAV collaborative system comprising a command acquisition unit, a command encoding unit, an environmental perception unit, an environmental encoding unit, a fusion encoding unit, a motion decoding unit, and an adaptive control unit. These units efficiently fuse command information and UAV sensor information, enabling the understanding of complex multimodal environmental information and complex collaborative commands in natural language form. Furthermore, no additional system is required for control command or environmental calibration, thus meeting the collaborative operation needs of UAVs in complex environments and expanding the application scenarios of UAVs. The multi-UAV collaborative control method allows multiple UAVs to access the collaborative system through adaptive control algorithms, adapting to the collaborative work needs of different aircraft types. Attached Figure Description

[0016] Figure 1 This is a system architecture diagram of a drone collaborative system based on embodied intelligence according to this application;

[0017] Figure 2 This is an information exchange diagram of a drone collaborative system based on embodied intelligence according to this application;

[0018] Figure 3 This is a flowchart of a multi-machine cooperative control method based on embodied intelligence according to this application. Detailed Implementation

[0019] The present application will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0020] This embodiment discloses a drone collaborative system based on embodied intelligence technology. Please refer to... Figure 1 It mainly includes an instruction acquisition unit A1, an instruction encoding unit A2, an environment perception unit A3, an environment encoding unit A4, a fusion encoding unit A5, an action decoding unit A6, and an adaptive control unit A7. Each unit will be described in detail below.

[0021] The instruction acquisition unit A1 is used to acquire voice instructions and text instructions.

[0022] The instruction encoding unit A2 is signal-connected to the instruction acquisition unit A1 and is used to preprocess the voice instructions and text instructions from the instruction acquisition unit A1, and obtain the first encoded information through instruction encoding.

[0023] The environmental sensing unit A3 is used to acquire multi-dimensional environmental sensing data of the UAV.

[0024] The environmental coding unit A4 is signal-connected to the environmental sensing unit A3 and is used to process multi-dimensional environmental sensing data from the environmental sensing unit A3, and obtain second coding information through data coding.

[0025] The fusion coding unit A5 is signal-connected to the instruction coding unit A2 and the environment coding unit A4, and is used to fuse the first coding information from the instruction coding unit A2 and the second coding information from the environment coding unit A4, and generate fused coding information through fusion.

[0026] The motion decoding unit A6 is signal-connected to the fusion coding unit A5 and is used to decode the fusion coding information from the fusion coding unit A5 to generate motion control commands for the UAV.

[0027] The adaptive control unit A7 is connected to the motion decoding unit signal A6 and is used to predict the motion state of the UAV based on the motion control command generated by the motion decoding unit A6 to obtain the state vector, and to perform motion control on the UAV according to the motion control command and the state vector.

[0028] In one specific embodiment, the instruction acquisition unit A1 includes a voice instruction module 11 and a text instruction module 12. The voice instruction module 11 and text instruction module 12 are integrated into a terminal controller for the UAV. The voice instruction module 11 is used to generate voice instructions by recognizing speech, and the text instruction module 12 is used to generate text instructions by recognizing text. It can be understood that the voice instruction module 11 and text instruction module 12 may be processor components that process speech and text, capable of acquiring work instructions in speech or text form and sending the instruction information to the instruction preprocessing module 18.

[0029] In one specific embodiment, the instruction encoding unit A2 includes an instruction preprocessing module 18 and a first LLM encoding module 19 integrated on the terminal controller for the UAV. The instruction preprocessing module 18 is used to convert voice instructions and text instructions into a word vector matrix, and the first LLM encoding module 19 is used to encode the word vector matrix and output first encoded information. It can be understood that the instruction preprocessing module 18 processes the voice and text signals from the instruction acquisition unit A1 into a word vector matrix, and then the first LLM encoding module 19 is responsible for feature encoding the word vector matrix.

[0030] In one specific embodiment, the environmental perception unit A3 includes a camera module 13, a laser point cloud module 14, and an attitude sensing module 15. These modules are integrated into the UAV, collecting corresponding sensing information on the UAV. For example, the camera module 13 generates RGB image data by photographing the environment in which the UAV is located; the laser point cloud module 14 generates laser point cloud data by scanning the environment with a laser; and the attitude sensing module 15 collects the UAV's altitude, tilt angle, speed, and position to generate flight attitude data. It can be understood that the camera module 13, laser point cloud module 14, and attitude sensing module 15 are respectively used to acquire RGB image data, laser point cloud data, and flight attitude data. These data are integrated into multi-dimensional environmental sensing data for transmission to the environmental preprocessing module 112.

[0031] In one specific embodiment, the environment coding unit A4 includes an environment preprocessing module 112 and a second LLM coding module 111 integrated on the terminal controller for the UAV. The environment preprocessing module 112 is used to convert multi-source environmental sensing data into a sensing data matrix, and the second LLM coding module 111 is used to encode the sensing data matrix and output second encoded information. This second encoded information has streaming characteristics, capable of preserving dynamic features between consecutive frames. It is understood that the environment coding unit A4 can transmit data with the environment sensing unit A3 via a wireless communication component, and the included environment preprocessing module 112 converts the sensing information from the environment sensing unit A3 into a sensing data matrix, while the second LLM coding module 111 is responsible for feature encoding the sensing data.

[0032] In one specific embodiment, the fusion encoding unit A5 is integrated into the terminal controller for the UAV, and uses hybrid encoding tokens to generate fusion encoding information from the fusion of the first encoding information and the second encoding information. It can be understood that the fusion encoding module A5 can fuse the encoding features from the instruction encoding unit A2 and the environment encoding unit A4, generate new hybrid encoding tokens, and send the generated new encoding to the multi-layer decoding module 113.

[0033] In one specific embodiment, the motion decoding unit A6 includes a multi-layer decoding module 113 and an encoding feedforward module 114 integrated on a terminal controller for a UAV. The multi-layer decoding module 113 is used to decode the fused encoded information into motion features, and the encoding feedforward module 114 is used to convert the motion features into executable motion control commands. It can be understood that the multi-layer decoding module 113 and the encoding feedforward module 114 are respectively responsible for encoding and decoding the hybrid features from the fused encoding module A5 into motion features and for converting the motion features into executable motion commands.

[0034] In one specific embodiment, the adaptive control unit A7 includes an end-to-end control module 16 and a state prediction module 17. The state prediction module 17 and the end-to-end control module 16 are integrated into the UAV. The state prediction module 17 predicts the UAV's motion state based on motion control commands to obtain a state vector. The end-to-end control module 16 performs motion control on the UAV based on its flight attitude data and state vector. It can be understood that the state prediction module 17 calculates the UAV's state vector, and the end-to-end control module 16 receives the UAV's attitude data and flight state vector to control its motion. The specific motion commands for the UAV come from the encoding feedforward module 114, enabling automatic adaptation to the characteristics of different UAVs and real-time adjustment of the control strategy to maintain UAV stability. Furthermore, the adaptive control unit A7 allows aircraft with various parameters to directly join the cooperative system, achieving stable control of the aircraft without the need for cumbersome prior parameter adjustments.

[0035] It should be noted that the instruction encoding unit A2, environment encoding unit A4, fusion encoding unit A5, and action decoding unit A6 are mainly integrated on the terminal controller for the UAV, and can be considered as the decision layer, responsible for processing various information or data and distributing control commands. Since the environment perception unit A3 and adaptive control unit A7 are mainly integrated on the UAV, they can be considered as the cluster layer, responsible for the UAV's sensing and status control.

[0036] Of course, the drone is treated as an intelligent agent 22, and the terminal controller for the drone is treated as the decision-making core 23, see details. Figure 2 The intelligent agent 22 can accept the environmental influences of its working environment and adapt to it through decision-making actions. For example, the drone collects information about its working environment through sensors to reflect environmental changes and adjusts its flight state after receiving control commands. The decision core 23 can also communicate with the intelligent agent 22, such as through long-distance wireless network communication, receiving collected information from the intelligent agent 22 and work commands from the user, and pushing action control commands to the intelligent agent. In essence, the intelligent agent 22 and the decision core 23 can interact. The intelligent agent 22 accepts the environmental influences of its working environment and influences the working environment through decision-making actions. The decision core 23 receives sensor information from the intelligent agent 22 and work commands from the user, and pushes state vectors and action commands to the intelligent agent 22. The action commands can control the spatial displacement and speed of the drone.

[0037] It should be noted that this application essentially introduces natural language command control logic into traditional UAV control. This requires the introduction of a large language model, enabling the UAV to acquire rich environmental information and task knowledge, thus improving its performance in complex scenarios. As a powerful deep learning model, the large language model possesses excellent representation and generalization capabilities. With the development of spatial intelligence technology, it has gained the ability to analyze and understand environmental characteristics. In the fusion and encoding of various information, the large language model can be used to dynamically adjust the UAV's motion state and behavioral strategies based on real-time environmental changes and task requirements, achieving more intelligent control. By integrating the large language model with multiple UAVs, and through an embodied intelligent decision-making core, the environment and inter-agent relationships are analyzed, and tasks are rationally allocated to the agents. This allows the intelligent unit to efficiently and accurately complete complex UAV collaborative tasks according to natural language commands, opening up new possibilities for the widespread application of UAVs. Furthermore, the large language model receives multimodal environmental data and control signals, such as images, LiDAR depth and attitude data, to infer and understand spatial information, environmental changes, and the relative relationships of UAVs. By combining the task requirements obtained from natural language instruction reasoning, corresponding action instructions can be assigned to all vehicles in the system, making it easy to achieve efficient collaborative work among multiple UAVs.

[0038] Understandable, based on Figure 1 The units in the system can efficiently integrate command information and UAV sensor information, understand complex multimodal environmental information and complex collaborative commands in natural language form, and do not require additional systems for control commands or environmental calibration. This meets the collaborative operation needs of UAVs in complex environmental conditions and expands the application scenarios of UAVs.

[0039] In one embodiment, based on the aforementioned drone cooperative system, this application also provides a multi-drone cooperative control method based on embodied intelligence. This method can be executed on a terminal controller (or decision core) for one or more drones. See details... Figure 3 .

[0040] Step S31 at the beginning of the algorithm is the entry point for algorithm execution.

[0041] Step S32 of the communication status check checks the communication connection status between all UAVs and the terminal controller (decision core) for the UAVs, records each UAV that has successfully connected to obtain the number of vehicles. The connection status is divided into two types: successful connection and connection error. The number of vehicles with a successful connection status is recorded.

[0042] Step S33, which determines the change in the number of connections, checks whether the number of successfully connected vehicles has changed. If it has changed, step S34, which involves scaling the IO tensor, is executed; otherwise, step S35, which involves updating the unit status, is executed.

[0043] In step S34 of IO tensor scaling, when the number of vehicles changes, the size of the tensor for receiving input information and buffering output data is adjusted according to the change in the number of vehicles; it can be understood that the size of the tensor for buffering input information and output data in the decision core is adjusted according to the change in the number of connections, and then the unit status update step S35 is executed.

[0044] Step S35 of the crew status update involves obtaining the flight status information and sensor status information of each successfully connected UAV, and updating the crew status vector based on the flight status information and sensor status information of each UAV. It can be understood that the crew status vector is updated based on the flight status and sensor status of the aircraft. The crew status vector is a one-dimensional array with a length equal to the number of successfully connected UAVs. The values ​​in the array represent the status of the corresponding aircraft, which are divided into aircraft health, sensor abnormality, controller abnormality, and flight status abnormality.

[0045] Step S36, which updates sensor information, involves acquiring corresponding multi-dimensional environmental sensor data from a drone in a normal state based on the crew state vector. This can be understood as reading sensor data from an aircraft whose sensor health is represented by the crew state vector. The sensor data can be RGB image data and 3D point cloud data.

[0046] In step S37 of the control signal update, voice commands and text commands are obtained; it can be understood that natural language control signals are read from the voice command module and the text command module.

[0047] In step S38 of input tensor fusion, the voice command and text command are encoded to obtain the first encoded information, and the multi-dimensional environmental sensing data are encoded to obtain the second encoded information. The first encoded information and the second encoded information are fused to generate fused encoded information. It can be understood that the input tensor is encoded by the environment encoding unit and the command encoding unit, and the environmental and command features are fused by the fusion encoding module.

[0048] In step S39 of the control output calculation, the fused encoded information is decoded to generate motion control commands for one or more UAVs; it can be understood that the motion output tensor is calculated by the motion decoding unit, and the motion output can be the three-dimensional displacement and motion velocity.

[0049] In step S310, which determines the valid output, the corresponding action output tensor is calculated based on the action control commands for one or more UAVs. It then determines whether each action output tensor exceeds a preset threshold. If so, the process proceeds to step S311, which distributes the control signal; otherwise, it proceeds to step S312, which sets the control signal to zero. Essentially, this step determines whether the magnitude of the output action is within a preset range.

[0050] In step S311 of the control signal distribution, when an action output tensor does not exceed the threshold, the corresponding action control command is sent to the corresponding UAV. It can be understood that this involves splitting the output action tensor and distributing the corresponding action to each UAV.

[0051] Step S312, which sets the control signal to zero, sets the corresponding action control command to zero when an action output tensor exceeds a threshold. This means that invalid output actions are set to zero, and the corresponding drone will temporarily stop moving.

[0052] Understandable, Figure 3 The multi-drone cooperative control method shown allows multiple UAVs to join the cooperative system through adaptive control algorithms, adapting to the needs of different aircraft types working together. Each UAV can be equipped with a parameter-free adaptive controller, enabling stable control of different UAV types. Any UAV can join the multi-drone cooperative system without additional parameter tuning, improving the operational efficiency and accuracy of UAV cooperation while reducing the skill and experience requirements of operators.

[0053] The above description, in conjunction with specific embodiments, provides a further detailed explanation of this application and should not be construed as limiting the specific implementation of this application to these descriptions. For those skilled in the art, several simple deductions or substitutions can be made without departing from the inventive concept of this application.

Claims

1. A collaborative unmanned aerial vehicle (UAV) system based on embodied intelligence, characterized in that, include: The instruction acquisition unit is used to acquire voice instructions and text instructions; The instruction encoding unit is signal-connected to the instruction acquisition unit and is used to preprocess the voice instruction and the text instruction, and to obtain first encoded information through instruction encoding. An environmental sensing unit is used to acquire diverse environmental sensing data from the drone. An environment coding unit is signal-connected to the environment sensing unit and is used to process the multi-dimensional environment sensing data and obtain second coding information through data coding. A fusion encoding unit, signal-connected to the instruction encoding unit and the environment encoding unit, is used to fuse the first encoded information from the instruction encoding unit and the second encoded information from the environment encoding unit, and generate fused encoded information through fusion; the fused encoded information uses hybrid encoded tokens; The motion decoding unit is signal-connected to the fusion encoding unit and is used to decode the fusion encoded information to generate motion control commands for the UAV. An adaptive control unit, signal-connected to the motion decoding unit, is used to predict the motion state of the UAV based on the motion control command to obtain a state vector, and to perform motion control on the UAV according to the motion control command and the state vector; The instruction encoding unit, the environment encoding unit, the fusion encoding unit, and the action decoding unit are integrated on the terminal controller for the UAV and are considered to be the decision layer, responsible for processing various information or data and distributing control instructions. The environmental perception unit and the adaptive control unit are integrated on the UAV and are considered to be the cluster layer, which is responsible for the UAV's sensing and status control.

2. The drone collaborative system based on embodied intelligence as described in claim 1, characterized in that, The command acquisition unit includes a voice command module and a text command module, which are integrated into a terminal controller for the UAV. The voice command module is used to generate the voice command by recognizing speech, and the text command module is used to generate the text command by recognizing text.

3. The drone collaborative system based on embodied intelligence as described in claim 2, characterized in that, The instruction encoding unit includes an instruction preprocessing module and a first LLM encoding module integrated on the terminal controller for the UAV; the instruction preprocessing module is used to convert the voice instruction and the text instruction into a word vector matrix, and the first LLM encoding module is used to encode the word vector matrix into instructions and output the first encoded information.

4. The drone collaborative system based on embodied intelligence as described in claim 1, characterized in that, The environmental perception unit includes a camera module, a laser point cloud module, and an attitude sensing module. The camera module, the laser point cloud module, and the attitude sensing module are integrated into the UAV. The camera module is used to generate RGB image data by capturing images of the environment in which the UAV is located. The laser point cloud module is used to generate laser point cloud data by scanning the environment in which the UAV is located with a laser. The attitude sensing module is used to collect the altitude, tilt angle, speed, and position of the UAV itself to generate flight attitude data. The RGB image data, the laser point cloud data, and the flight attitude data are integrated to form the multi-dimensional environmental sensing data.

5. The drone collaborative system based on embodied intelligence as described in claim 4, characterized in that, The environment coding unit includes an environment preprocessing module and a second LLM coding module integrated on the terminal controller for the UAV. The environmental preprocessing module is used to convert the multivariate environmental sensing data into a sensing data matrix, and the second LLM encoding module is used to encode the sensing data matrix and output the second encoding information. The second encoded information has streaming characteristics and can preserve the dynamic features between consecutive frames.

6. The drone collaborative system based on embodied intelligence as described in claim 1, characterized in that, The adaptive control unit includes an end-to-end control module and a state prediction module; the state prediction module and the end-to-end control module are integrated into the UAV. The state prediction module is used to predict the motion state of the UAV based on the motion control command to obtain a state vector. The end-to-end control module performs motion control on the UAV according to the motion control command and the state vector.

7. The drone collaborative system based on embodied intelligence as described in any one of claims 1-6, characterized in that, The drone is considered an intelligent agent, and the terminal controller for the drone is considered the decision-making core. The intelligent agent is able to accept the environmental influences of its working environment and adapt to the working environment through decision-making actions. The decision-making core is able to communicate with the intelligent agent, receive collected information from the intelligent agent and work instructions from the user, and push action control instructions to the intelligent agent.

8. A multi-machine cooperative control method based on embodied intelligence, characterized in that, include: Check the communication connection status of all drones and the terminal controller for the drones, and record each drone that successfully connects to obtain the number of vehicles. When the number of vehicles changes, the size of the tensor for receiving input information and buffering output data is adjusted according to the change in the number of vehicles. Acquire the flight status information and sensor status information of each successfully connected UAV, and update the group status vector based on the flight status information and sensor status information of each UAV. Based on the unit state vector, obtain the corresponding multi-dimensional environmental sensing data from the UAV in normal state; Get voice and text commands; The voice command and the text command are encoded to obtain first encoded information, and the multi-dimensional environmental sensing data are encoded to obtain second encoded information. The first encoded information and the second encoded information are fused to generate fused encoded information; The fused encoded information uses hybrid encoded tokens; The fused encoded information is decoded to generate motion control commands for one or more of the UAVs; Calculate the corresponding motion output tensor based on the motion control commands for one or more of the UAVs, and determine whether each motion output tensor exceeds a preset threshold. When an action output tensor exceeds the threshold, the corresponding action control command is set to zero; otherwise, the corresponding action control command is sent to the corresponding UAV.

Images