A collaborative decision-making control method, device, equipment and medium for multiple unmanned aerial vehicles

Abstract

The present application provides a collaborative decision-making control method, device, equipment and medium for multiple drones, including: inputting the mission information of multiple drones into a deep reinforcement learning model, performing information interaction processing on the mission information of multiple drones based on multiple Actor networks in the deep reinforcement learning model to predict the next-moment decision information of each drone; processing the next-moment decision information of each drone based on a position controller and an attitude controller to generate a next-moment control command for each drone; calculating the resistance and ground effect force generated by the propeller of each drone, adjusting the current attitude information of the drone based on the resistance and ground effect force to determine the adjusted information, and feeding the adjusted information back to the deep reinforcement learning model to continue to predict the action information of each drone at future moments, thereby improving the adaptive ability and decision-making autonomy level of the drone in complex environments.

Description

Technical Field

[0001] The present application relates to the field of drone control technology, and in particular to a collaborative decision-making control method, device, equipment and medium for multiple drones. Background Art

[0002] With the rapid development of UAV technology, multi-UAV systems have shown broad application prospects in military, agriculture, logistics and other fields. However, there are many challenges in realizing collaborative decision-making control of multi-UAV systems, especially in dynamic and complex environments. At present, the collaborative decision-making control methods of multi-UAVs mainly include decision-making methods based on expert knowledge and intelligent optimization algorithms. The theory based on expert knowledge is suitable for situations where the combat environment is simple and the prior experience is clear, and has the advantage of fast decision-making speed. However, this method has weak adaptability to the environment and needs to rely on a large number of rules to assist decision-making. In addition, under normal circumstances, expert knowledge is difficult to obtain. When the expert knowledge base lacks relevant templates, the decision accuracy will be significantly reduced. Intelligent optimization algorithms such as genetic algorithms, particle swarm algorithms, ant colony algorithms and pigeon swarm algorithms, etc., can reduce search complexity, improve search efficiency and meet real-time requirements by using bionic characteristics. These algorithms solve problems by simulating evolution and group behavior in nature. However, these algorithms are more suitable for simple environments. When the environment is complex, the search efficiency of these algorithms will drop significantly, and they are prone to fall into local optimal solutions and lack global optimization capabilities. Therefore, how to accurately realize the collaborative decision-making control of multiple UAVs has become a technical issue that cannot be underestimated. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a collaborative decision-making control method, device, equipment and medium for multiple UAVs, to realize collaborative decision-making control of multiple UAV systems through a deep reinforcement learning model, to improve the adaptability and decision-making autonomy of UAVs in complex environments, and to use position controllers and attitude controllers to generate specific control commands based on decision information to achieve precise control of multiple UAVs.

[0004] The present application provides a collaborative decision-making control method for multiple UAVs, the collaborative decision-making control method comprising:

[0005] Determine the mission information of multiple drones based on the drone mission scenario; wherein the mission information includes observation information, execution action information and environment information;

[0006] Inputting the task information of the plurality of drones into a pre-trained deep reinforcement learning model, capturing the interaction information of the task information between the plurality of drones based on the gated recurrent units, soft attention mechanism and graph neural network of the plurality of actor networks in the deep reinforcement learning model, and predicting the next moment decision information of each drone;

[0007] Processing the next moment decision information of each UAV based on the position controller and the attitude controller to generate the next moment control command of each UAV;

[0008] In the process of adjusting the speed and direction of the control motor based on the control command of each drone at the next moment, the resistance and ground effect force generated by the propeller of each drone are calculated, and the current posture information of the drone is adjusted based on the resistance and the ground effect force to determine the adjusted information, and the adjusted information is fed back to the deep reinforcement learning model to continue to predict the action information of each drone at the future moment.

[0009] In a possible implementation, for the task information of each drone, the task information of the multiple drones is input into a pre-trained deep reinforcement learning model, and the gated recurrent units, soft attention mechanisms, and graph neural networks of multiple Actor networks in the deep reinforcement learning model are used to capture the interaction information of the task information between the multiple drones, and predict the next moment decision information of each drone, including:

[0010] Inputting the mission information of the drone into the gated recurrent unit of the Actor network, performing time series processing on the state features in the mission information of the drone, and generating a hidden state feature vector of the drone at the current moment;

[0011] Based on the soft attention mechanism, the hidden state feature vector of the current moment of the UAV and the hidden state feature vectors of other UAVs are processed to determine the attention weight between the UAV and other UAVs;

[0012] The graph neural network aggregates the hidden state feature vectors of other drones at the current moment based on the attention weights between the drone and other drones, and determines the aggregated hidden state feature vectors;

[0013] Performing vector splicing on the aggregated hidden state feature vector and the hidden state feature vector of the drone at the current moment to generate a spliced ​​hidden state feature vector, and processing the spliced ​​hidden state feature vector based on a multi-layer perceptron to generate action information of the drone at the next moment;

[0014] The action information at the next moment is converted and processed to generate the decision information of the drone at the next moment.

[0015] In a possible implementation manner, the performing information conversion processing on the action information at the next moment to generate the decision information of the drone at the next moment includes:

[0016] Integrating the acceleration information in the next moment decision information to determine the target speed information in the next moment decision information;

[0017] The target speed information is integrated to determine the target position information in the next moment decision information.

[0018] In a possible implementation manner, for each of the drones, the position controller and the attitude controller process the next moment decision information of each of the drones to generate the next moment control command of each of the drones, including:

[0019] The position controller determines the target attitude information and target thrust in the control command at the next moment based on the current position information and current speed information of the drone, the target position information, the target speed information and the target yaw angle information in the next moment decision information;

[0020] The attitude controller determines the control torque in the control command at the next moment based on the current attitude information, current angular velocity information, the target attitude information and the target angular velocity information in the next moment decision information of the drone.

[0021] In a possible implementation, the deep reinforcement learning model is trained by the following steps:

[0022] Inputting sample task information and reward functions of multiple sample drones in the sample drone task scene into the initial deep reinforcement learning model, processing sample state features in the sample task information of multiple sample drones based on multiple Actor networks, and predicting the next moment action information of each of the sample drones;

[0023] Determine the reward value corresponding to the next moment action information of each sample drone based on the reward function;

[0024] Input the next moment action information of each sample drone and the new state information corresponding to the next moment action information of the sample drone into the corresponding Critic network, process the next moment action information and the new state information, and generate the state action value;

[0025] The network parameters of the corresponding Critic network are updated based on the loss value between the state action value and the reward value, and the network parameters of the Actor network are updated based on the policy gradient until the number of iterations is reached and the updating of the network parameters of the Actor network and the Critic network is stopped to determine the deep reinforcement learning model.

[0026] In a possible implementation manner, the next moment action information of each sample drone and the new state information corresponding to the sample drone when executing the next moment action information are input into the Critic network, and the next moment action information and the new state information are processed to generate a state action value, including:

[0027] Characteristic encoding is performed on the next moment action information and the new state information of the sample drone to determine the action feature corresponding to the next moment action information and the new state feature corresponding to the new state information;

[0028] Multi-head attention processing is performed on the action features of the next moment after feature encoding and the new state features, and nonlinear transformation processing is performed on the action features of the next moment after feature encoding and the new state features after the multi-head attention processing based on a multi-layer perceptron to generate the state-action value.

[0029] In a possible implementation, the resistance generated by the propeller of the drone is determined by the following formula:

[0030]

[0031] in, is the propeller speed, For height, is a constant, is the linear speed, i For drones, For resistance.

[0032] The embodiment of the present application also provides a collaborative decision-making control device for multiple UAVs, the collaborative decision-making control device comprising:

[0033] A scenario task definition module, used to determine the task information of multiple drones based on the drone task scenario; wherein the task information includes observation information, execution action information and environment information;

[0034] A decision-making module, which is used to input the task information of the plurality of drones into a pre-trained deep reinforcement learning model, and to capture the interactive information of the task information between the plurality of drones based on the gated recurrent units, soft attention mechanism and graph neural network of the plurality of actor networks in the deep reinforcement learning model, and predict the decision information of each drone at the next moment;

[0035] A control module, used for processing the next moment decision information of each UAV based on the position controller and the attitude controller, and generating the next moment control command of each UAV;

[0036] The UAV environment platform is used to calculate the resistance and ground effect force generated by the propeller of each UAV during the process of adjusting the speed and direction of the control motor based on the next control command of each UAV, adjust the current posture information of the UAV based on the resistance and the ground effect force to determine the adjusted information, and feed the adjusted information back to the deep reinforcement learning model to continue to predict the action information of each UAV at the future moment.

[0037] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the collaborative decision-making control method for multiple drones as described above are performed.

[0038] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the collaborative decision-making control method for multiple unmanned aerial vehicles as described above are executed.

[0039] The embodiments of the present application provide a collaborative decision-making control method, device, equipment and medium for multiple drones, the collaborative decision-making control method comprising: determining the task information of multiple drones based on the drone task scenario; wherein the task information includes observation information, execution action information and environmental information; inputting the task information of the multiple drones into a pre-trained deep reinforcement learning model, capturing the interaction information of the task information between the multiple drones based on the gated recurrent units, soft attention mechanism and graph neural network of the multiple Actor networks in the deep reinforcement learning model, and predicting the next moment decision information of each drone; processing the next moment decision information of each drone based on the position controller and the attitude controller to generate the next moment control command of each drone; in the process of adjusting the speed and direction of the control motor based on the next moment control command of each drone, calculating the resistance and ground effect force generated by the propeller of each drone, adjusting the current attitude information of the drone based on the resistance and the ground effect force to determine the adjusted information, and feeding the adjusted information back to the deep reinforcement learning model to continue to predict the action information of each drone at the future moment. Through the deep reinforcement learning model, collaborative decision-making and control of multi-UAV systems can be achieved, the adaptability and decision-making autonomy of UAVs in complex environments can be improved, and the position controller and attitude controller can be used to generate specific control commands based on the decision information to achieve precise control of multiple UAVs.

[0040] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A flowchart of a collaborative decision-making control method for multiple UAVs provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of a deep reinforcement learning model provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a collaborative decision-making control method for multiple UAVs provided in an embodiment of the present application;

[0045] Figure 4 This is a simulation result diagram of a multi-UAV cooperative reconnaissance mission;

[0046] Figure 5 One of the structural schematic diagrams of a collaborative decision-making control device for multiple UAVs provided in an embodiment of the present application;

[0047] Figure 6 The second structural diagram of a collaborative decision-making control device for multiple UAVs provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0050] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied in the field of drone control technology.

[0051] Research has found that with the rapid development of drone technology, multi-drone systems have shown broad application prospects in military, agriculture, logistics and other fields. However, there are many challenges in realizing collaborative decision-making control of multi-drone systems, especially in dynamic and complex environments. At present, the collaborative decision-making control methods of multi-drone systems mainly include decision-making methods based on expert knowledge and intelligent optimization algorithms. The theory based on expert knowledge is suitable for situations where the combat environment is simple and the prior experience is clear, and has the advantage of fast decision-making speed. However, this method has weak adaptability to the environment and needs to rely on a large number of rules to assist decision-making. In addition, under normal circumstances, expert knowledge is difficult to obtain. When the expert knowledge base lacks relevant templates, the decision accuracy will be significantly reduced. Intelligent optimization algorithms such as genetic algorithms, particle swarm algorithms, ant colony algorithms and pigeon colony algorithms, etc., can reduce search complexity, improve search efficiency and meet real-time requirements by using bionic characteristics. These algorithms solve problems by simulating evolution and group behavior in nature. However, these algorithms are more suitable for simple environments. When the environment is complex, the search efficiency of these algorithms will be significantly reduced, and they are prone to fall into local optimal solutions and lack global optimization capabilities. Therefore, how to accurately achieve collaborative decision-making and control of multiple UAVs has become a technical issue that cannot be underestimated.

[0052] Based on this, an embodiment of the present application provides a collaborative decision-making control method for multiple UAVs, which realizes collaborative decision-making control of multiple UAV systems through a deep reinforcement learning model, improves the adaptability and decision-making autonomy of UAVs in complex environments, and uses position controllers and attitude controllers to generate specific control commands based on decision information to achieve precise control of multiple UAVs.

[0053] See also Figure 1 , Figure 1 This is a flow chart of a collaborative decision-making control method for multiple drones provided in an embodiment of the present application. Figure 1 As shown in , the collaborative decision control method provided by the embodiment of the present application includes:

[0054] S101: Determine mission information of multiple drones based on drone mission scenarios; wherein the mission information includes observation information, execution action information, and environment information.

[0055] In this step, mission information of multiple drones is determined according to the drone mission scenario.

[0056] Among them, the observation information specifically includes position, altitude, speed, acceleration, direction, other drone information, target location, obstacle information, sensor input, etc. Execution action information specifies the set of actions that the drone can perform, such as adjusting the flight attitude (pitch, roll, yaw), changing the flight speed, lifting, etc. Determine the dimension and range of the action space to ensure that the drone has flexibility in a dynamic environment. Environmental information such as wind speed, obstacle location, flight area boundaries, etc. Clarify safety constraints, such as maximum flight altitude, no-fly zones, etc. Reward function, used to evaluate the task completion of the multi-drone system and provide feedback for reinforcement learning algorithm training. The reward function includes: Immediate reward defines an immediate reward for each step of action and provides real-time feedback. The immediate reward can be dynamically adjusted during the flight based on sensor data. Terminal reward sets the terminal reward based on the task completion or failure condition. High rewards are given for successful completion of the task, and large penalties are given for task failure or violation of safety constraints.

[0057] S102: Input the task information of the multiple drones into a pre-trained deep reinforcement learning model, capture the interactive information of the task information between the multiple drones based on the gated recurrent units, soft attention mechanism and graph neural network of the multiple Actor networks in the deep reinforcement learning model, perform information interaction processing on the task information of the multiple drones, and predict the next moment decision information of each drone.

[0058] In this step, the mission information of multiple drones is input into a pre-trained deep reinforcement learning model. The gated recurrent units, soft attention mechanism, and graph neural network of multiple Actor networks in the deep reinforcement learning model are used to capture the interactive information of the mission information between multiple drones, and the next-moment decision information of each drone is predicted.

[0059] Among them, the next moment decision information includes target speed information, target position information, target yaw angle information, etc.

[0060] Here, the deep reinforcement learning model is the MACDPG algorithm, which can handle complex state space and action space and adapt to complex and changing environments. The decision-making process learns the optimal decision-making strategy by training a large number of historical mission scenarios, ensuring that drones can adaptively allocate tasks, avoid obstacles and work collaboratively. The output of the algorithm is the specific execution decision information for each drone, including the acceleration or speed of the drone in three dimensions.

[0061] In a possible implementation, for the task information of each drone, the task information of the multiple drones is input into a pre-trained deep reinforcement learning model, and the gated recurrent units, soft attention mechanisms, and graph neural networks of multiple Actor networks in the deep reinforcement learning model are used to capture the interaction information of the task information between the multiple drones, and predict the next moment decision information of each drone, including:

[0062] A: Input the mission information of the drone into the gated recurrent unit of the Actor network, perform time series processing on the state features in the mission information of the drone, and generate the hidden state feature vector of the drone at the current moment.

[0063] Here, the drone’s mission information is input into the gated recurrent unit of the Actor network, and the state features in the drone’s mission information are processed in time series to generate the drone’s current hidden state feature vector.

[0064] For each drone i Status characteristics , use the GRU unit to process its own state features and obtain the hidden state vector of the agent : , is the hidden state feature of the previous moment, It is the drone of the moment i The hidden state features of .

[0065] B: Based on the soft attention mechanism, the hidden state feature vector of the current moment of the UAV and the hidden state feature vectors of the current moment of other UAVs are processed to determine the attention weight between the UAV and other UAVs.

[0066] Here, the hidden state feature vector of the drone at the current moment and the hidden state feature vectors of other drones at the current moment are processed according to the soft attention mechanism to determine the attention weight between the drone and the other drones.

[0067] Among them, the Soft-Attention weight is calculated for each drone It is necessary to calculate the hidden state features of the drone at the current moment The current hidden state characteristics of all other drones The similarity between the UAV and other UAVs is used to generate the attention weight. ,in, is the dimension of the hidden state. Attention weight Representing an Agent For Agents attention.

[0068] C: The graph neural network aggregates the current hidden state feature vectors of other drones based on the attention weights between the drone and other drones to determine the aggregated hidden state feature vectors.

[0069] Here, the graph neural network aggregates the current hidden state feature vectors of other drones based on the attention weights between the drone and other drones, and determines the aggregated hidden state feature vector.

[0070] Among them, by calculating the drone For all other drones The attention weights aggregate the hidden states of neighbors in an additive way: , It's a drone The aggregated hidden state feature vector with other drones.

[0071] D: Concatenate the aggregated hidden state feature vector and the hidden state feature vector of the drone at the current moment to generate a concatenated hidden state feature vector, and process the concatenated hidden state feature vector based on a multi-layer perceptron to generate the next moment action information of the drone.

[0072] Here, the aggregated hidden state feature vector and the hidden state feature vector of the drone at the current moment are vector-concatenated to generate a concatenated hidden state feature vector, and the concatenated hidden state feature vector is processed according to the multi-layer perceptron to generate the action information of the drone at the next moment.

[0073] Among them, the aggregated With its own hidden state Splicing, ,symbol Represents the concatenation operation of the vector. Finally, a multi-layer perceptron (MLP) is used to generate the final action. , Here it is indicated The next action information output by the policy network (Actor network).

[0074] E: Perform information conversion processing on the action information at the next moment to generate the decision information of the drone at the next moment.

[0075] Here, the action information at the next moment is converted and processed to generate the decision information of the drone at the next moment.

[0076] In a possible implementation manner, the performing information conversion processing on the action information at the next moment to generate the decision information of the drone at the next moment includes:

[0077] The acceleration information in the next moment decision information is integrated to determine the target speed information in the next moment decision information; the target speed information is integrated to determine the target position information in the next moment decision information.

[0078] Here, the acceleration signal output by the reinforcement learning model is effectively converted into a motion trajectory, including speed and position. Its core function is to convert acceleration data into continuous speed and position data through numerical integration to achieve precise motion control. At each time step of the system, the speed value is calculated by integrating the acceleration. Specifically, the speed of the current time step is is the velocity of the previous time step Add the acceleration of the current time step Multiply by the time step Calculated. The formula is as follows: Next, we will calculate the speed As input, it is integrated again to obtain the position information of the current time step This process is done by increasing the speed Multiply by the time step Add to the position of the previous time step To achieve this. The formula is: , starting from the initialization phase, by setting the initial position and velocity, receiving the acceleration signal from the reinforcement learning model, and calculating the new velocity and position using the above integral formula in each time step. The final calculated velocity and position values ​​are used to update the state of the drone, providing accurate data support for subsequent time steps. This method ensures a smooth transition from acceleration signals to motion trajectories, optimizing the drone's motion control and analysis process.

[0079] In this application, the Actor network first uses a gated recurrent unit (GRU) to process time series data, capture the temporal dependencies between states, and generate a hidden layer representation of the current state. Secondly, the soft-attention mechanism is used to weight the hidden layer representation to highlight the features that have an important impact on the current decision, thereby improving the accuracy of the decision. The interactive information between multiple drones is captured through a graph neural network (GNN) to generate a joint feature representation of each drone. This representation integrates the relationship between multiple agents and helps to better coordinate decision-making. Then, the features of each drone are spliced ​​through feature concatenation (CAT) to form a complete feature vector, which contains the self-information and aggregated information of all drones. Finally, the spliced ​​feature vector is nonlinearly transformed through a multi-layer perceptron (MLP) to generate the decision action of each drone. The Actor network generates actions according to the current strategy and applies these actions to the environment. Multiple Actor networks work together to generate decision information for different drones.

[0080] In a possible implementation, the deep reinforcement learning model is trained by the following steps:

[0081] a: Input the sample task information and reward function of multiple sample drones in the sample drone task scene into the initial deep reinforcement learning model, process the sample state features in the sample task information of multiple sample drones based on multiple Actor networks, and predict the next moment action information of each of the sample drones.

[0082] Here, the sample task information and reward function of multiple sample drones in the sample drone task scene are input into the initial deep reinforcement learning model. The sample state features in the sample task information of multiple sample drones are processed according to multiple Actor networks to predict the next moment action information of each sample drone.

[0083] Among them, during the training process, the Actor network processes the sample state characteristics and predicts that the execution steps of the next moment action information of each sample drone are consistent with the execution steps of the next moment action information of the above-mentioned drone, which will not be repeated in this part.

[0084] b: Determine the reward value corresponding to the next moment action information of each sample drone based on the reward function.

[0085] Here, the reward value corresponding to the next moment action information of each sample drone is determined according to the reward function.

[0086] Among them, the reward functions corresponding to different scene tasks are different. If the scene task is intelligent obstacle avoidance, the corresponding reward function R is constructed through three components to promote the drone to reach the target point efficiently and safely. These three components include: , The reward value is inversely proportional to the Euclidean distance between the target landmark and the nearest drone, which means that the closer the best performing drone is to the target, the higher the reward value is, so as to encourage multiple drones to explore the target as quickly as possible and approach the target. It indicates the reward for reaching the target point. If the drone reaches the vicinity of the target point, a large reward will be given to encourage the drone to stay at the target point after reaching it. This section deals with the collision between the drone and obstacles. If the drone avoids a collision, this part of the reward is zero or a positive value; if a collision occurs, a negative reward, i.e. a penalty, will be given. The purpose of this setting is to avoid unsafe behavior of the drone.

[0087] c: Input the next moment action information of each sample drone and the new state information corresponding to the next moment action information of the sample drone into the corresponding Critic network, process the next moment action information and the new state information, and generate the state action value.

[0088] Here, the next moment action information of each sample drone and the new state information corresponding to the next moment action information of the sample drone are input into the corresponding Critic network, and the next moment action information and the new state information are processed to generate the state action value.

[0089] Here, the Critic network evaluates the value and provides feedback based on the actions and current status generated by the Actor network. This feedback is used to guide the strategy update of the Actor network to improve the effectiveness of decision-making. Through collaborative training between multiple agents, the Critic and Actor networks continuously optimize their respective decision-making strategies. During the training process, drones learn from each other to improve the performance of the overall system.

[0090] In a possible implementation manner, the next moment action information of each sample drone and the new state information corresponding to the sample drone when executing the next moment action information are input into the Critic network, and the next moment action information and the new state information are processed to generate a state action value, including:

[0091] I: feature encoding the next moment action information and new state information of the sample drone, and determining the action feature corresponding to the next moment action information and the new state feature corresponding to the new state information.

[0092] Here, feature encoding is performed on the next moment action information and new state information of the sample drone to determine the action features corresponding to the next moment action information and the new state features corresponding to the new state information.

[0093] II: Perform multi-head attention processing on the action features of the next moment after feature encoding and the new state features, and perform nonlinear transformation processing on the action features of the next moment after feature encoding and the new state features after multi-head attention processing based on a multi-layer perceptron to generate the state-action value.

[0094] Here, multi-head attention processing is performed on the action features of the next moment after feature encoding and the new state features, and nonlinear transformation processing is performed on the action features of the next moment after feature encoding after multi-head attention processing and the new state features according to the multi-layer perceptron to generate state-action values.

[0095] Among them, the Critic network first inputs the state features and action features of the next moment into the Emb network, performs a feature encoding dimension reduction, and then uses the multi-head attention mechanism module to capture the important features in the input state and action data. Through multiple attention heads, the input features are processed in parallel to enhance the model's sensitivity and capture ability to state and action information. Then, the features output by the attention mechanism are nonlinearly transformed through MLP to generate state action values, and the value of state actions is evaluated for subsequent guidance of the Actor network's strategy update. Multiple Critic networks work together to evaluate the state action values ​​of different agents.

[0096] d: Based on the loss value between the state-action value and the reward value, the network parameters of the corresponding Critic network are updated, and based on the policy gradient, the network parameters of the Actor network are updated until the number of iterations is reached, and the network parameters of the Actor network and the Critic network are stopped from being updated, so as to determine the deep reinforcement learning model.

[0097] Here, the network parameters of the corresponding Critic network are updated according to the loss value between the state action value and the reward value, and the network parameters of the Actor network are updated based on the policy gradient until the number of iterations is reached, and the updating of the network parameters of the Actor network and the Critic network is stopped to determine the deep reinforcement learning model.

[0098] Among them, the initial state of the entire drone system , where N is the number of drones, for each drone Initialize the policy network (Actor network) Parameters and the value network (Critic network) Parameters . Constructing the graph ,node Assemble for drones ,side Represents the interaction between drones and creates an experience playback buffer . From the buffer Randomly sample a batch of experience, update the value network, and generate a , calculate the target action , target action , calculate the target value , , For drones i The reward value is minimized to update , loss function Update the policy network for each drone , use policy gradient to update policy network parameters Policy Gradient Update , using soft update parameters Update the target network: , .

[0099] Here, in this application, the deep reinforcement learning model needs to be trained for different scenario tasks so that the deep reinforcement learning model can predict decision information for different scenario tasks.

[0100] For further information, see Figure 2 , Figure 2 This is a schematic diagram of a deep reinforcement learning model provided in an embodiment of the present application. Figure 2 As shown, the first UAV mission letter The input is processed into the gated recurrent unit of the action network to obtain the hidden state features corresponding to the current moment of the drone. , the hidden state features corresponding to the current moment of the drone The hidden state features of other drones at the current moment are input into the soft attention mechanism to obtain the current moment t The attention weight between the next drone and other drones , the attention weight Input into the graph convolution to aggregate the hidden state features of other drones to obtain the aggregated hidden state features , the hidden state features and the hidden state features at the current moment The input is sent to the splicing layer for splicing processing, and the spliced ​​features are sent to the fully connected layer for processing to obtain the action information of the drone at the next moment. , the action information and task information The state features and action features are input into the embedding layer of the value network for processing to obtain state features and action features. The state features and action features are input into the multi-head attention mechanism for processing. The processed state features and action features are input into the fully connected layer to generate the state-action value V. The state-action value V is used to adjust the parameters of the corresponding action network and value network until the number of iterations is reached and the parameter adjustment is stopped.

[0101] S103: Processing the next moment decision information of each UAV based on the position controller and the attitude controller to generate the next moment control command of each UAV.

[0102] In this step, the next moment decision information of each UAV is processed according to the position controller and the attitude controller to generate the next moment control command of each UAV.

[0103] In a possible implementation manner, for each of the drones, the position controller and the attitude controller process the next moment decision information of each of the drones to generate the next moment control command of each of the drones, including:

[0104] (1): The position controller determines the target attitude information and target thrust in the control command at the next moment based on the current position information, current speed information of the UAV, the target position information, target speed information and target yaw angle information in the next moment decision information.

[0105] Here, the position controller determines the target attitude information and target thrust in the control command at the next moment based on the current position information, current speed information of the drone, the target position information, target speed information and target yaw angle information in the next moment decision information.

[0106] (2): The attitude controller determines the control torque in the control command at the next moment based on the current attitude information, current angular velocity information, the target attitude information and the target angular velocity information in the next moment decision information of the UAV.

[0107] Here, the attitude controller determines the control torque in the control command at the next moment based on the current attitude information, current angular velocity information, target attitude information and target angular velocity information in the next moment decision information of the drone.

[0108] In a specific embodiment, a PID controller is used to achieve precise control of the motor speed of the drone to achieve a predetermined position and attitude target. The position controller receives the target position , Target speed and target yaw angle The input, combined with the current position and speed , calculate the target posture information and target thrust The attitude controller then receives these target attitudes and angular velocity , and combined with the current posture and angular velocity Calculate the required control torque After that, the motor controller will and target thrust Adjust the motor speed , and finally drives the drone to reach the predetermined position and posture.

[0109] In this application, the next moment decision information output by the decision module is converted into control instructions. The control module includes some execution control units. The linear one is implemented by PID, and the direct operation is directly converted. The key to this step is to define accurate PID parameters for the parameters of the drone to achieve stable control of the drone.

[0110] S104: In the process of adjusting the speed and direction of the control motor based on the control command of each drone at the next moment, the resistance and ground effect force generated by the propeller of each drone are calculated, and the current posture information of the drone is adjusted based on the resistance and the ground effect force to determine the adjusted information, and the adjusted information is fed back to the deep reinforcement learning model to continue to predict the action information of each drone at the future moment.

[0111] In this step, the speed and direction of the control motor are adjusted according to the control command of each UAV at the next moment, and the resistance and ground effect force generated by the propeller of each UAV are calculated. The current posture information of the UAV is adjusted according to the resistance and ground effect force to determine the adjusted information, and the adjusted information is fed back to the deep reinforcement learning model to continue to predict the action information of each UAV in the future.

[0112] In a possible implementation, the resistance generated by the propeller of the drone is determined by the following formula:

[0113]

[0114] in, is the propeller speed, For height, is a constant, is the linear speed, i For drones, For resistance.

[0115] When a quadcopter is hovering at very low altitudes, ground effect occurs, which causes the propeller airflow to interact with the ground, increasing thrust. as follows:

[0116]

[0117] in, is the propeller radius, is the propeller speed, For height, is a constant.

[0118] Here, in a heterogeneous UAV system composed of multiple UAVs with different functions and performances, their diversity and complementarity give them significant advantages in complex tasks. For example, fixed-wing UAVs can perform large-scale reconnaissance and long-term cruising, while multi-rotor UAVs are suitable for close observation and fine operation. The collaborative decision-making control method of multiple UAVs in this application can enable the heterogeneous UAV system to effectively divide the work and flexibly respond to dynamic environments, so that heterogeneous UAVs can perform adaptive task allocation and path planning in dynamic environments.

[0119] For further information, see Figure 3 , Figure 3 A schematic diagram of a collaborative decision-making control method for multiple UAVs provided in an embodiment of the present application. Figure 3 As shown, the task information and reward function in the scene task are input into the reinforcement learning algorithm in the decision module for processing, and the action information of the drone at the next moment is obtained. The action information of the drone at the next moment is processed to obtain the decision information of the drone at the next moment, and the decision information at the next moment is input into the control module. The position execution control unit, attitude execution control unit and firing execution control unit are used to process and obtain the control instructions, and the control instructions are sent to the drone environment platform so that the propeller of the drone can produce different lift and thrust distribution according to the speed and direction of different motors, thereby adjusting the attitude of the entire drone on three axes (yaw, pitch, roll) and realizing displacement in space. Then the perception information of the next moment is generated and handed over to the decision module to loop the execution process to finally complete the task.

[0120] In the specific implementation, the execution functions of each module and the information flow relationship in the collaborative decision-making control framework of multiple UAVs are as follows: First, the effective extraction and definition of task information in the framework is the basis for determining the effectiveness of subsequent decisions. According to the specific task scenario, the necessary observation information and environmental features are extracted, and the corresponding action space is constructed. Subsequently, a reward function is designed based on this information, and the design of the function includes the incentives for positive behaviors and the penalties for negative behaviors to ensure that the UAV can continuously optimize its behavior strategy during the learning process. Accurate task parameter definition and reasonable reward function setting are the core of this step. Second, the construction of the decision module mainly relies on the improved multi-UAV collaborative deep reinforcement learning algorithm Multi-Agent Combination Deep Policy Gradient (MACDPG) based on the Actor-Critic (AC) framework. The module receives task information, observation information and reward function as input, and uses MACDPG to calculate and output the corresponding collaborative decision action. At the same time, the information processing module converts the collaborative decision action into the target control amount required by the control module and transmits it to the control module. The design of the MACDPG algorithm enables the UAV to make decisions based on historical information, and can interact and cooperate with other UAVs to achieve more optimized collaborative decision-making. Third, the main function of the control module is to process the output of the decision module and obtain the lowest-level control instructions. This module includes multiple execution control units, such as using a PID controller to achieve linear control, or real-time control of directly operable instructions. Fourth, the multi-UAV environment platform implemented based on Pybullet controls the operation of the UAV motor by receiving the control instructions of the control module, thereby controlling the speed of each propeller of the UAV and controlling the flight of the UAV. The lift and thrust distribution generated by this operation determines the attitude of the UAV in space (yaw, pitch and roll), and simulates various aerodynamic effects such as ground effect and downwash effect based on Pybullet to achieve accurate displacement feedback. At the same time, the perception information collected by the UAV during the process will also be synchronously fed back to the decision module to form a closed loop to achieve continuous execution and optimization of the task.

[0121] In the specific embodiment, for the multi-UAV cooperative reconnaissance mission, this mission aims to use multiple UAVs to autonomously allocate tasks and intelligently avoid obstacles to adapt to complex urban environments and diverse mission requirements, and improve the efficiency and flexibility of mission execution. The mission area is the air and the surrounding areas of the city, including other aircraft, dangerous areas and natural obstacles. These elements may affect the flight path and operation of the UAV. A complex UAV autonomous navigation mission is designed, the goal is to let three UAVs make autonomous decisions to fly to three randomly set target points through reinforcement learning. The initial position of each UAV is randomly generated within a preset range, and their task is to safely arrive at the designated location in an environment with multiple obstacles. The obstacles are eight spherical obstacles, each with a radius of 0.25, and the positions are randomly distributed. The collision is a soft collision. When a collision occurs, the UAV will not stop, and the position of the target landmark is also random. This setting is intended to simulate various uncertainties and complex situations that UAVs may encounter in the real world, and test the navigation and decision-making capabilities of the reinforcement learning algorithm in a multi-target, dynamic environment. The action space of the experiment is defined as the three-dimensional acceleration of the UAV, that is, each step of the reinforcement learning algorithm determines the acceleration of each UAV in three spatial dimensions. The observation space includes the speed of the UAV itself and location , the locations of the three target landmarks , 16 sensor signals of the drone in the horizontal direction , Drone diagonally above 8 sensor signals for one week , Drone diagonally below 8 sensor signals for one week , a sensor signal directly above the drone and a sensor signal directly below the drone The action space represents the acceleration of the drone in three-dimensional space, where represents the acceleration in the x-axis direction, represents the acceleration in the y-axis direction, Represents the acceleration in the z-axis direction, positive values ​​represent the positive direction, and negative values ​​represent the negative direction. The reward function R is constructed through three components to promote the drone to reach the target point efficiently and safely. These three components include:

[0122]

[0123] The part is inversely proportional to the Euclidean distance between the target landmark and the nearest UAV, where the current position of the UAV is , No. The target point location is , which means that the closer the best performing drone is to the target, the higher the reward value will be, so as to encourage multiple drones to explore the target as quickly as possible and approach it.

[0124]

[0125] It indicates the reward for reaching the target point. If the drone reaches the vicinity of the target point, a large reward will be given to encourage the drone to stay at the target point after reaching it.

[0126]

[0127] This section deals with the collision between the drone and obstacles. If the drone avoids a collision, this part of the reward is zero or a positive value; if a collision occurs, a negative reward, i.e. a penalty, will be given. The purpose of this setting is to avoid unsafe behavior of the drone.

[0128]

[0129] For further information, see Figure 4 , Figure 4 The simulation results of multi-UAV cooperative reconnaissance mission are shown in the figure below. Figure 4 As shown in the figure, the big sphere represents the randomly generated obstacles in the air, and the small sphere represents the destination. The drone starts from a random position in the lower left corner, makes decisions through the reinforcement learning model, and executes actions through the PID control model to achieve the effect of automatically allocating task target points and avoiding obstacles to reach the destination.

[0130] In this application, its parameters and corresponding reward functions can be set for different mission scenarios. Through the deep reinforcement learning algorithm, it can self-learn to adapt to high-dynamic scenarios and unknown environments. Through the PID control unit, the decision information output by the reinforcement learning is converted into control instructions to achieve direct control of the drone. The drone simulation environment built based on Pybullet can support the aerodynamic effects of the real environment and realize the dynamic update of the state of the drone's non-particle dynamics model. This framework can make full use of the advantages of deep reinforcement learning, and can make corresponding decision actions based on the current state in real time online to adapt to the complex and changeable real environment. In addition, the framework overcomes the lack of mature adaptive frameworks and the limitations of experiments based on particle models in existing research. It can effectively integrate the needs of different tasks, apply deep reinforcement learning to the control level, and ensure that drones can achieve efficient collaborative decision-making and control in actual deployment and operation, and significantly improve their adaptability in dynamic environments.

[0131] An embodiment of the present application provides a collaborative decision-making control method for multiple drones, the collaborative decision-making control method comprising: determining mission information of multiple drones based on drone mission scenarios; wherein the mission information includes observation information, execution action information and environmental information; inputting the mission information of the multiple drones into a pre-trained deep reinforcement learning model, and performing information interaction processing on the mission information of the multiple drones based on multiple Actor networks in the deep reinforcement learning model to predict the next-moment decision information of each drone; processing the next-moment decision information of each drone based on a position controller and an attitude controller to generate a next-moment control command for each drone; in the process of adjusting the speed and direction of the control motor based on the next-moment control command of each drone, calculating the resistance and ground effect force generated by the propeller of each drone, adjusting the current attitude information of the drone based on the resistance and the ground effect force to determine the adjusted information, and feeding the adjusted information back to the deep reinforcement learning model to continue to predict the action information of each drone at a future moment. Through the deep reinforcement learning model, collaborative decision-making and control of multi-UAV systems can be achieved, the adaptability and decision-making autonomy of UAVs in complex environments can be improved, and the position controller and attitude controller can be used to generate specific control commands based on the decision information to achieve precise control of multiple UAVs.

[0132] See also Figure 5 , Figure 6 , Figure 5 One of the structural schematic diagrams of a collaborative decision-making control device for multiple UAVs provided in an embodiment of the present application; Figure 6 This is a second structural diagram of a multi-UAV collaborative decision-making control device provided in an embodiment of the present application. Figure 5 As shown in , the collaborative decision control device 500 includes:

[0133] A scenario task definition module 510 is used to determine the task information of multiple drones based on the drone task scenario; wherein the task information includes observation information, execution action information and environment information;

[0134] A decision module 520 is used to input the task information of the plurality of drones into a pre-trained deep reinforcement learning model, and based on the gated recurrent units, soft attention mechanism and graph neural network of the plurality of Actor networks in the deep reinforcement learning model, capture the interactive information of the task information between the plurality of drones, and predict the next moment decision information of each drone;

[0135] The control module 530 is used to process the next moment decision information of each drone based on the position controller and the attitude controller to generate the next moment control command of each drone;

[0136] The UAV environment platform 540 is used to calculate the resistance and ground effect force generated by the propeller of each UAV during the process of adjusting the speed and direction of the control motor based on the next control command of each UAV, adjust the current posture information of the UAV based on the resistance and the ground effect force to determine the adjusted information, and feed the adjusted information back to the deep reinforcement learning model to continue to predict the action information of each UAV at the future moment.

[0137] Furthermore, the decision module 520 is used for inputting the task information of the plurality of drones into a pre-trained deep reinforcement learning model for the task information of each drone, and capturing the interaction information of the task information between the plurality of drones based on the gated recurrent units, soft attention mechanism and graph neural network of the plurality of Actor networks in the deep reinforcement learning model, and predicting the next moment decision information of each drone. The decision module 520 is specifically used for:

[0138] Inputting the mission information of the drone into the gated recurrent unit of the Actor network, performing time series processing on the state features in the mission information of the drone, and generating a hidden state feature vector of the drone at the current moment;

[0139] Based on the soft attention mechanism, the hidden state feature vector of the current moment of the UAV and the hidden state feature vectors of other UAVs are processed to determine the attention weight between the UAV and other UAVs;

[0140] The graph neural network aggregates the hidden state feature vectors of other drones at the current moment based on the attention weights between the drone and other drones, and determines the aggregated hidden state feature vectors;

[0141] Performing vector splicing on the aggregated hidden state feature vector and the hidden state feature vector of the drone at the current moment to generate a spliced ​​hidden state feature vector, and processing the spliced ​​hidden state feature vector based on a multi-layer perceptron to generate action information of the drone at the next moment;

[0142] The action information at the next moment is converted and processed to generate the decision information of the drone at the next moment.

[0143] Furthermore, when the decision module 520 is used to perform information conversion processing on the action information at the next moment to generate the decision information of the drone at the next moment, the decision module 520 is specifically used to:

[0144] Integrating the acceleration information in the next moment decision information to determine the target speed information in the next moment decision information;

[0145] The target speed information is integrated to determine the target position information in the next moment decision information.

[0146] Further, when the control module 530 is used for processing the next moment decision information of each drone based on the position controller and the attitude controller to generate the next moment control command of each drone, the control module 530 is specifically used for:

[0147] The position controller determines the target attitude information and target thrust in the control command at the next moment based on the current position information and current speed information of the drone, the target position information, the target speed information and the target yaw angle information in the next moment decision information;

[0148] The attitude controller determines the control torque in the control command at the next moment based on the current attitude information, current angular velocity information, the target attitude information and the target angular velocity information in the next moment decision information of the drone.

[0149] Further, such as Figure 6 As shown, the collaborative decision control device 500 further includes a model training module 550, which is used to train the deep reinforcement learning model through the following steps:

[0150] Inputting sample task information and reward functions of multiple sample drones in the sample drone task scene into the initial deep reinforcement learning model, processing sample state features in the sample task information of multiple sample drones based on multiple Actor networks, and predicting the next moment action information of each of the sample drones;

[0151] Determine the reward value corresponding to the next moment action information of each sample drone based on the reward function;

[0152] Input the next moment action information of each sample drone and the new state information corresponding to the next moment action information of the sample drone into the corresponding Critic network, process the next moment action information and the new state information, and generate the state action value;

[0153] The network parameters of the corresponding Critic network are updated based on the loss value between the state action value and the reward value, and the network parameters of the Actor network are updated based on the policy gradient until the number of iterations is reached and the updating of the network parameters of the Actor network and the Critic network is stopped to determine the deep reinforcement learning model.

[0154] Furthermore, when the model training module 550 is used to input the next moment action information of each sample drone and the new state information corresponding to the sample drone when executing the next moment action information into the Critic network, and to process the next moment action information and the new state information to generate the state action value, the model training module 550 is specifically used to:

[0155] Characteristic encoding is performed on the next moment action information and the new state information of the sample drone to determine the action feature corresponding to the next moment action information and the new state feature corresponding to the new state information;

[0156] Multi-head attention processing is performed on the action features of the next moment after feature encoding and the new state features, and nonlinear transformation processing is performed on the action features of the next moment after feature encoding and the new state features after the multi-head attention processing based on a multi-layer perceptron to generate the state-action value.

[0157] Furthermore, the drone environment platform 540 determines the resistance generated by the propeller of the drone by the following formula:

[0158]

[0159] in, is the propeller speed, For height, is a constant, is the linear speed, i For drones, For resistance.

[0160] The embodiment of the present application provides a collaborative decision-making control device for multiple drones, the collaborative decision-making control device comprising: a scenario task definition module, used to determine the task information of multiple drones based on the drone task scenario; wherein the task information includes observation information, execution action information and environmental information; a decision module, used to input the task information of multiple drones into a pre-trained deep reinforcement learning model, based on the gated recurrent units, soft attention mechanism and graph neural network of multiple Actor networks in the deep reinforcement learning model to capture the interaction information of the task information between multiple drones, and predict the next moment decision information of each drone; a control module, used to process the next moment decision information of each drone based on a position controller and an attitude controller to generate a next moment control command for each drone; a drone environment platform, used to calculate the resistance and ground effect force generated by the propeller of each drone in the process of adjusting the speed and direction of the control motor based on the next moment control command of each drone, adjust the current attitude information of the drone based on the resistance and the ground effect force to determine the adjusted information, and feed the adjusted information back to the deep reinforcement learning model to continue to predict the action information of each drone at a future moment. Through the deep reinforcement learning model, collaborative decision-making and control of multi-UAV systems can be achieved, the adaptability and decision-making autonomy of UAVs in complex environments can be improved, and the position controller and attitude controller can be used to generate specific control commands based on the decision information to achieve precise control of multiple UAVs.

[0161] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown in , the electronic device 700 includes a processor 710 , a memory 720 and a bus 730 .

[0162] The memory 720 stores machine-readable instructions executable by the processor 710. When the electronic device 700 is running, the processor 710 communicates with the memory 720 via the bus 730. When the machine-readable instructions are executed by the processor 710, the above-mentioned Figure 1 The steps of the collaborative decision-making control method for multiple UAVs in the method embodiment shown in the figure, and the specific implementation method can be found in the method embodiment, which will not be repeated here.

[0163] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1The steps of the collaborative decision-making control method for multiple UAVs in the method embodiment shown in the figure, and the specific implementation method can be found in the method embodiment, which will not be repeated here.

[0164] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0165] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0166] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0168] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0169] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A collaborative decision-making control method for multiple UAVs, characterized in that: The collaborative decision-making control method comprises: Determine the mission information of multiple drones based on the drone mission scenario; wherein the mission information includes observation information, execution action information and environment information; Inputting the task information of the plurality of drones into a pre-trained deep reinforcement learning model, capturing the interaction information of the task information between the plurality of drones based on the gated recurrent units, soft attention mechanism and graph neural network of the plurality of actor networks in the deep reinforcement learning model, and predicting the next moment decision information of each drone; Processing the next moment decision information of each UAV based on the position controller and the attitude controller to generate the next moment control command of each UAV; In the process of adjusting the speed and direction of the control motor based on the control command of each drone at the next moment, the resistance and ground effect force generated by the propeller of each drone are calculated, and the current posture information of the drone is adjusted based on the resistance and the ground effect force to determine the adjusted information, and the adjusted information is fed back to the deep reinforcement learning model to continue to predict the action information of each drone at the future moment.

2. The collaborative decision-making control method according to claim 1, characterized in that: For the task information of each drone, the task information of the multiple drones is input into a pre-trained deep reinforcement learning model, and the gated recurrent units, soft attention mechanisms, and graph neural networks of multiple Actor networks in the deep reinforcement learning model are used to capture the interactive information of the task information between the multiple drones, and predict the next moment decision information of each drone, including: Inputting the mission information of the drone into the gated recurrent unit of the Actor network, performing time series processing on the state features in the mission information of the drone, and generating a hidden state feature vector of the drone at the current moment; Based on the soft attention mechanism, the hidden state feature vector of the current moment of the UAV and the hidden state feature vectors of other UAVs are processed to determine the attention weight between the UAV and other UAVs; The graph neural network aggregates the hidden state feature vectors of other drones at the current moment based on the attention weights between the drone and other drones, and determines the aggregated hidden state feature vectors; Performing vector splicing on the aggregated hidden state feature vector and the hidden state feature vector of the drone at the current moment to generate a spliced ​​hidden state feature vector, and processing the spliced ​​hidden state feature vector based on a multi-layer perceptron to generate action information of the drone at the next moment; The action information at the next moment is converted and processed to generate the decision information of the drone at the next moment.

3. The collaborative decision-making control method according to claim 2, characterized in that: The step of performing information conversion processing on the action information at the next moment to generate the decision information of the drone at the next moment includes: Integrating the acceleration information in the next moment decision information to determine the target speed information in the next moment decision information; The target speed information is integrated to determine the target position information in the next moment decision information.

4. The collaborative decision-making control method according to claim 1, characterized in that: For each of the drones, the position controller and the attitude controller process the next moment decision information of each of the drones to generate the next moment control command of each of the drones, including: The position controller determines the target attitude information and target thrust in the control command at the next moment based on the current position information and current speed information of the drone, the target position information, the target speed information and the target yaw angle information in the next moment decision information; The attitude controller determines the control torque in the control command at the next moment based on the current attitude information, current angular velocity information, the target attitude information and the target angular velocity information in the next moment decision information of the drone.

5. The collaborative decision-making control method according to claim 1, characterized in that: The deep reinforcement learning model is trained by the following steps: Inputting sample task information and reward functions of multiple sample drones in the sample drone task scene into the initial deep reinforcement learning model, processing sample state features in the sample task information of multiple sample drones based on multiple Actor networks, and predicting the next moment action information of each of the sample drones; Determine the reward value corresponding to the next moment action information of each sample drone based on the reward function; Input the next moment action information of each sample drone and the new state information corresponding to the next moment action information of the sample drone into the corresponding Critic network, process the next moment action information and the new state information, and generate the state action value; The network parameters of the corresponding Critic network are updated based on the loss value between the state action value and the reward value, and the network parameters of the Actor network are updated based on the policy gradient until the number of iterations is reached and the updating of the network parameters of the Actor network and the Critic network is stopped to determine the deep reinforcement learning model.

6. The collaborative decision-making control method according to claim 5, characterized in that: The next moment action information of each sample drone and the new state information corresponding to the sample drone when executing the next moment action information are input into the Critic network, and the next moment action information and the new state information are processed to generate the state action value, including: Characteristic encoding is performed on the next moment action information and the new state information of the sample drone to determine the action feature corresponding to the next moment action information and the new state feature corresponding to the new state information; Multi-head attention processing is performed on the action features of the next moment after feature encoding and the new state features, and nonlinear transformation processing is performed on the action features of the next moment after feature encoding and the new state features after the multi-head attention processing based on a multi-layer perceptron to generate the state-action value.

7. The collaborative decision-making control method according to claim 1, characterized in that: The drag generated by the propeller of the drone is determined by the following formula: in, For the i The propeller speed of each drone, is a constant, is the linear speed, For resistance.

8. A collaborative decision-making control device for multiple UAVs, characterized in that: The collaborative decision-making control device comprises: A scenario task definition module, used to determine the task information of multiple drones based on the drone task scenario; wherein the task information includes observation information, execution action information and environment information; A decision-making module, which is used to input the task information of the plurality of drones into a pre-trained deep reinforcement learning model, and to capture the interactive information of the task information between the plurality of drones based on the gated recurrent units, soft attention mechanism and graph neural network of the plurality of actor networks in the deep reinforcement learning model, and predict the decision information of each drone at the next moment; A control module, used for processing the next moment decision information of each UAV based on the position controller and the attitude controller, and generating the next moment control command of each UAV; The UAV environment platform is used to calculate the resistance and ground effect force generated by the propeller of each UAV during the process of adjusting the speed and direction of the control motor based on the next control command of each UAV, adjust the current posture information of the UAV based on the resistance and the ground effect force to determine the adjusted information, and feed the adjusted information back to the deep reinforcement learning model to continue to predict the action information of each UAV at the future moment.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the collaborative decision-making control method for multiple unmanned aerial vehicles as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the collaborative decision-making control method for multiple UAVs as described in any one of claims 1 to 7 are executed.

Images