Multi-agent unmanned aerial vehicle search task energy optimization method based on entropy maximization strategy

Abstract

The invention discloses a multi-agent unmanned aerial vehicle search task energy optimization method based on an entropy maximization strategy, and belongs to the field of unmanned aerial vehicle search task energy optimization. The implementation method comprises the following steps: modeling a multi-region search task of the multi-agent unmanned aerial vehicle, constructing an energy consumption model of the unmanned aerial vehicle, taking the minimum energy consumption of a multi-agent system as a priority target, adopting a reinforcement learning algorithm with maximized multi-agent entropy, making a decision on the action of each unmanned aerial vehicle, and obtaining a multi-region search task of the multi-agent unmanned aerial vehicle. According to the method, the randomness of strategy implementation of each agent is enhanced, premature falling into a suboptimal solution is avoided, real-time adjustment is performed on the next action decision of each unmanned aerial vehicle according to the environment and energy consumption observed by other unmanned aerial vehicles based on a method of centralized training in distributed execution, and when the unmanned aerial vehicle arrives at a target area, the unmanned aerial vehicle is subjected to real-time control. And carrying out full-path search on the target area to ensure the optimality of solving sub-problems, namely, realizing energy optimization of a multi-unmanned aerial vehicle search task based on an entropy maximization strategy.

Description

Technical Field

[0001] The present invention belongs to the field of energy optimization of unmanned aerial vehicle search tasks, and relates to an energy optimization method for multi-agent unmanned aerial vehicle search tasks based on an entropy maximization strategy. Background Art

[0002] The multi-agent search task problem refers to the problem of using multiple autonomous agents (such as drones, robots, etc.) to effectively cover the entire area or part of the area within a specified area. The goal of coverage path planning is to design the movement path of each agent so that the entire area is fully and evenly covered, while minimizing overlapping coverage between agents and avoiding collisions.

[0003] For this type of problem, past studies often regard it as a combination of the traveling salesman problem (TSP) and the covering path planning (CPP). These methods usually have the problems of high computational cost in the offline phase and poor observation ability in the online phase. When dealing with multi-agent problems, it is difficult to dynamically adjust the strategy and apply it to problems with energy constraints. Summary of the invention

[0004] The purpose of the present invention is to provide an energy optimization method for a multi-agent UAV search task based on an entropy maximization strategy, to model the multi-agent UAV search task in multiple areas, to construct an energy consumption model for the UAV, to prioritize the minimum energy consumption of the multi-agent system, to adopt a multi-agent entropy maximization reinforcement learning algorithm, to make decisions on the actions of each UAV, to enhance the randomness of each agent's strategy, to avoid falling into a suboptimal solution too early, and based on a centralized training and distributed execution method, to make real-time adjustments to the next action decision of each UAV according to the environment and energy consumption observed by other UAVs, and when the UAV reaches its target area, to conduct a full path search for the target area to ensure the optimality of solving subproblems, that is, to achieve energy optimization of multi-UAV search tasks based on an entropy maximization strategy.

[0005] The purpose of the present invention is achieved through the following technical solutions.

[0006] The multi-agent UAV search task energy optimization method of the entropy maximization strategy disclosed in the present invention comprises the following steps:

[0007] Step 1: Create a rasterized map of size K×K×K, determine and define the initial positions of n drones in the Cartesian coordinate system, and use m spatially distributed unconnected polygons as the target area U of the search task j , constraining the maximum search angle θ of a single UAV and the maximum effective search height d of the UAV max , assign the target area to n drones, and dispatch the drones to cover m polygonal areas to reach Uj All coordinate points in space are searched by the drone.

[0008] n drones are distributed on the rasterized map, and the initial position of the drone in the Cartesian coordinate system is c i (i=1,2,...n), each drone moves at a constant speed. For a gridded map of size K×K×K, the state space of the drone is represented as S;

[0009] S = {loc x}×{loc y}×{loc z}

[0010] Among them, loc x ,loc y ,loc z ∈[0,K], representing c i The x, y, z coordinates of the position in the Cartesian coordinate system.

[0011] Target area: There are m spatially distributed disconnected polygonal areas represented as:

[0012] U j (j=1,2,...,m)

[0013] Each region consists of a set of coordinate points close to the ground composition;

[0014] The search angle of a single drone is θ, and the maximum effective search height of the drone is d max , when the coordinates of a certain drone are (x i ,y i ,z i ), the drone can effectively search for the coordinates (x, y, z) of the point:

[0015]

[0016] (zz i ) 2 <d max 2

[0017] In the grid map, based on the UAV search angle constraint and the UAV search maximum height constraint, the target area is assigned to n UAVs, and the UAVs are dispatched to cover m polygonal areas to meet U j All coordinate points in space are effectively searched by the drone.

[0018] Step 2: Based on the equivalent area S of the drone body F , Inductive power factor κp , blade drag coefficient σ e 、Blade resistance coefficient Ω e , rotation radius R e , air density ρ, rotor solidity R S 、UAV weight M UAV Establishing the UAV power propulsion model P F (V), and based on the UAV power driving model, the UAV energy consumption model E is established i (t).

[0019] Drone Power Propulsion Model P F (V) is expressed as:

[0020]

[0021] Among them, S F represents the equivalent area of ​​the fuselage, κ p represents the induced power factor, g represents the acceleration of gravity, σ e represents the blade drag coefficient, Ω e represents the blade drag coefficient, R e represents the rotation radius, ρ represents the air density, R S Indicates rotor solidity, M UAV Indicates the weight of the drone. v0 represents the average rotor speed during hovering, d0 is the fuselage drag ratio, s is the rotor solidity, V represents the flight speed, and A represents the propeller area;

[0022] The drone is in the time span T H Energy consumption E i (t) is expressed as:

[0023] E i (t) = P i T (t)T H

[0024] Since each drone maintains a constant flying speed, there is a positive correlation between the energy consumption of a single drone and the flight distance.

[0025] Step 3: Define the drone state space U i , based on the drone’s information S i , the observed environment information FOV, and the information of the target area T iThe action space A of the drone in the map is defined. A reward function is defined to ensure the collision safety of the drone and optimize the total energy consumption. Based on the reward function, the drone is guaranteed to have collision safety and optimize the total energy consumption during the mission. The reward function includes the heading reward of the drone. Collision Penalty between Drones Penalty for drone crossing the boundary or touching the ground Energy consumption penalty for drones to complete missions Energy depletion penalty during drone missions Rewards for all drones reaching their targets

[0026] The state space is the information S of the drone i , the observed environment information FOV, and the information of the target area T i A collection of;

[0027] U i =(S i ,FOV i ,E S ,T i )

[0028] in, id represents the serial number of the drone, pos represents the position of the drone in the Cartesian coordinate system, γ represents the deflection angle of the drone, and C∈{idle,busy,depleted} represents the current state of the drone, which is idle, working, and energy depleted. E S Indicates the remaining energy of the drone, expressed as:

[0029]

[0030] Among them, E i 0 represents the initial energy of UAV i; t is the continuous working time;

[0031] T i The unit vector pointing from the drone to the next area The Euclidean distance l from the drone to the target is composed of:

[0032]

[0033] At time t, through the action a of drone i i t Information about the drone i t , calculate the information S of the drone at the next moment i t+1;

[0034]

[0035] By updating S i Get the state space at each moment;

[0036] The discrete action space of the drone is represented as

[0037] A={turn left ,turn right ,rise,decline,maintain}

[0038] turn left Indicates that the drone turns left, turn right The drone deflects to the right, rise means the drone is rising, decline means the drone is descending, and maintain means the drone maintains the current horizontal direction and moves in a straight line; L means left, the right deflection angle is the same; the rise and fall angles are the same;

[0039] The reward function of the drone includes the following parts: Expressed as Collision Penalty between Drones Defined as parameter p1, the penalty for the drone crossing the boundary or touching the ground Defined as parameter p2, the energy consumption penalty of the drone under the condition of completing the task Expressed as -βE max -E s |, Energy depletion penalty during drone missions Defined as parameter p3, the reward for all drones reaching the target Defined as parameter p4.

[0040] in, Representation vector Angle in Cartesian coordinate system; γ t Represents the angle of the vector from the current position of the drone to the target point; α and β are constants; based on the reward function, the drone is guaranteed to have collision safety and optimize total energy consumption during the mission.

[0041] Step 4: Add an entropy factor based on the reinforcement learning objective function, construct an entropy maximization strategy to make decisions on the drone's actions, and improve the drone's randomness and exploration capabilities when making decisions.

[0042] Construct an entropy maximization strategy to make decisions on the actions of the drone. The specific implementation method is:

[0043] Initialize the policy network π of each drone iθ , local value network And combined into a joint value function, the Critic network is optimized by minimizing the time difference loss, and the factorized soft policy iteration method is used to optimize the policy network by maximizing the policy entropy and the joint value function. The expectation of the joint value function is a linear combination of the value functions and entropies of i agents. The joint strategy is introduced to obtain an objective function that combines the joint value and entropy terms and decomposes the complex joint value function into a form that can be optimized. During training, the entropy maximization policy network is optimized through iterative interaction, so that the entropy maximization strategy gradually approaches the optimal.

[0044] The reward function of the entropy maximization method adds an entropy factor to the reward function in step 3 to enhance the randomness of the decision-making of the drone's actions, so that the drone can explore more for the next action, increase the randomness of its strategy, and avoid obtaining a local optimal solution. The objective function after adding the entropy factor is:

[0045]

[0046] Among them, s t represents the state at time t, a t represents the action at time t, ρ π is the marginal condition of the state-action distribution, r(s t ,a t ) represents the state s at a certain moment t Next, perform action a t The reward obtained, H(π(a t |s t )) is the entropy of the strategy, α is the entropy temperature coefficient, which can be equivalent to the weight coefficient of the entropy factor and the original reward function; Indicates that under a given strategy π, state s t and action a t According to the state-action distribution ρ π The expected value calculated.

[0047] Among them, entropy is used to characterize the probability distribution of each drone action, and its expression is as follows:

[0048] H=log(π θ (a'|τ'))

[0049] π θ (a'|τ') means that given the information τ', the strategy π θ The probability of selecting action a', whose action a' is included in the action space A in step 3.

[0050] Initialize the policy network π of each drone i θ, local value network And combined into a joint value function:

[0051]

[0052] Among them, g ω It is a single-layer monotone mixture network.

[0053] The maximum value of the joint value function is consistent with the maximum value function of a single drone

[0054]

[0055] Sampling and interaction: The drone interacts with the environment based on the current strategy π θ Select action a t After executing the action, the environment feedbacks the new state s t+1 and instant rewards t The interaction data is stored in the experience pool for subsequent training, and the interaction data includes state, action, reward, and new state.

[0056] Evaluation strategy, optimizes the Critic network by minimizing the temporal difference loss. The difference loss is expressed as follows:

[0057]

[0058] in, is the current joint value function, y tot is the target value of the value function, y tot The calculated values ​​are as follows:

[0059]

[0060]

[0061] in are the parameters of the target Critic, Actors and hybrid network respectively, and s is the generation bias of the input.

[0062] Policy update uses a factorized soft policy iteration method to optimize the policy network by maximizing the policy entropy and the joint value function:

[0063]

[0064] The expectation of the joint value function is a linear combination of the value functions and entropy of the i agents, where ω i (s) is the weight, b(s) is the bias

[0065]

[0066] Introducing a joint strategy The entropy terms of each agent are summarized into an overall entropy expression for further calculation:

[0067]

[0068] An objective function is obtained that combines the joint value and entropy terms and decomposes the complex joint value function into a form that can be optimized, so that during training, the policy network is optimized by maximizing this objective function.

[0069]

[0070] Target network update: Regularly perform soft updates on the target network to enhance the stability of training:

[0071] φ'←τφ+(1-τ)φ'

[0072] Where τ is the soft update coefficient.

[0073] Repeat the above steps of evaluating strategy, updating strategy, and updating target network. Through multiple iterative interactive optimizations, the entropy maximization strategy gradually approaches the optimal.

[0074] Step 5: Introduce the drone energy model in step 2, calculate the energy consumption and remaining stored energy of each drone strategy selection (state switching), use the priority planning strategy to plan the target area for each drone, and make decisions on the drone's actions based on the entropy maximization strategy in step 4. At the same time, consider the situation where the drone energy is exhausted and the drone cannot complete the task, reallocate the drone tasks, and achieve energy optimization of multi-agent drone search tasks.

[0075] Use the priority planning strategy to calculate the target area list l for each drone t Priority, l t Expressed as

[0076] l t =[U1,U2...U n ]

[0077] When each drone performs a mission, its action decision is obtained by the entropy maximization strategy of the four multi-agents in step 4. Compared with the completely independent multi-agent algorithm, the multi-agent entropy maximization strategy increases the randomness of drone decisions by introducing the system entropy term, avoids overload and falls into suboptimal solutions, and improves the overall performance of the intelligent system.

[0078] Considering the situation where the drone is unable to complete the task due to exhaustion of energy, the energy optimization of the multi-agent drone search task is realized. The control system returns its target number to the task list and reallocates the available drones;

[0079]

[0080] When the drone arrives at the sub-area, it will apply for a full coverage search inside the sub-area. After the drone completes the task, the drone's status will change from busy to idle, and the drone in the idle state will be rescheduled for a new task.

[0081] Beneficial effects:

[0082] 1. The present invention discloses an energy optimization method for a multi-agent UAV search task. By using an entropy maximization algorithm, entropy is introduced as one of the indicators of the evaluation function. By maximizing entropy, the randomness of the agent's decision-making is increased, and a larger state-action space is explored, which helps to find the optimal strategy in a complex environment and avoid falling into a suboptimal solution too early, thereby enhancing the robustness of the strategy in the face of environmental changes or noise. In the optimization process, the weight of the entropy term is dynamically adjusted through the entropy temperature coefficient to balance the relationship between exploration and utilization, thereby ensuring the efficiency of task execution while avoiding the negative impact of excessive randomization of the strategy.

[0083] 3. The present invention discloses an energy optimization method for a multi-agent UAV search task. During the strategy update process, a factorized soft strategy iteration method is used to decompose the joint action value function into a linear combination of the critic value and entropy of each agent, thereby optimizing the strategy network, improving the efficiency of strategy optimization, and achieving better collaboration in the multi-agent system.

[0084] 2. The present invention discloses an energy optimization method for a multi-agent UAV search task. The method constructs an energy consumption model of a quad-rotor UAV, calculates the energy consumption of the UAV in performing the task, and uses energy consumption as a core indicator in the reward function, thereby achieving the optimal energy effect. At the same time, by considering the total energy constraint of each agent in performing the task, it provides response measures when a single agent is exhausted of energy or an emergency occurs, thereby ensuring the completion of the UAV search task. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 Flowchart of the energy optimization method for multi-agent UAV search tasks based on the entropy maximization strategy of the present invention. DETAILED DESCRIPTION

[0086] like Figure 1 As shown, the multi-agent UAV search task energy optimization method of the entropy maximization strategy disclosed in this embodiment has the following specific implementation steps:

[0087] Step 1: Problem Description:

[0088] Step 1.1 Distribute 10 drones on the rasterized map. The initial position of the drone in the Cartesian coordinate system is c i (i=1,2,...10), each drone moves at a constant speed. For a rasterized map of size 200×200×200, the state space of the drone is represented as S;

[0089] S = {loc x}×{loc y}×{loc z}

[0090] Among them, loc x ,loc y ,loc z ∈[0,K], representing c i The x, y, z coordinates of the position in the Cartesian coordinate system.

[0091] Target area: There are 30 spatially distributed disconnected polygonal areas represented as:

[0092] U j (j=1,2,...,30)

[0093] Each region consists of a set of coordinate points close to the ground composition;

[0094] Step 1.2 The search angle of a single drone is θ, and the maximum effective search height of the drone is d max , when the coordinates of a certain drone are (x i ,y i ,z i ), the drone can effectively search for the coordinates (x, y, z) of the point only if the following conditions are met:

[0095]

[0096] (zz i ) 2 <d max 2

[0097] The task of this embodiment is to assign the target area to n drones and dispatch the drones to cover m polygonal areas so that j All coordinate points in space are effectively searched by the drone;

[0098] Step 2: Drone Energy Consumption Model

[0099] Step 2.1 Drone power push model P F (V) is expressed as:

[0100]

[0101] Among them, S F represents the equivalent area of ​​the fuselage, κ p represents the induced power factor, g represents the acceleration of gravity, σ e represents the blade drag coefficient, Ω e represents the blade drag coefficient, R e represents the rotation radius, ρ represents the air density, R S Indicates rotor solidity, M UAV Indicates the weight of the drone. v0 represents the average rotor speed during hovering, d0 is the fuselage drag ratio, s is the rotor solidity, V represents the flight speed, and A is the symbol explanation;

[0102] The drone is in the time span T H Energy consumption E i (t) is expressed as:

[0103] E i (t) = P i T (t)T H

[0104] Since each drone maintains a constant flight speed, there is a positive correlation between the energy consumption of a single drone and the flight distance;

[0105] Step 3: Define the state space, action space, and reward function

[0106] Step 3.1 The state space is the information S of each group of drones i , the observed environment information FOV, and the information of the target area T i A collection of;

[0107] U i =(S i ,FOV i ,E S ,T i )

[0108] in, id represents the serial number of the drone, pos represents the position of the drone in the Cartesian coordinate system, γ represents the deflection angle of the drone, and C∈{idle,busy,depleted} represents the current state of the drone, which is idle, working, and depleted. E S Indicates the remaining energy of the drone, expressed as:

[0109]

[0110] represents the initial energy of UAV i; t is the continuous working time;

[0111] T i The unit vector pointing from the drone to the next area The Euclidean distance l from the drone to the target is composed of:

[0112]

[0113] At time t, through the action a of drone i i t Information about the drone i t , calculate the information S of the drone at the next moment i t+1 ;

[0114]

[0115] By updating S i Get the state space at each moment;

[0116] Step 3.2 The discrete action space of the drone is represented as

[0117] A={turn left ,turn right ,rise,decline,maintain}

[0118] turn left Indicates that the drone turns left, turn right The drone deflects to the right, rise means the drone is rising, decline means the drone is descending, and maintain means the drone maintains the current horizontal direction and moves in a straight line; L means left, the right deflection angle is the same; the rise and fall angles are the same;

[0119] Step 3.3 Reward Function

[0120]

[0121] When the drone is heading to the target area, its reward function is set according to the standards in the table. Representation vector Angle in Cartesian coordinate system; γ t The angle of the vector from the current position of the drone to the target point; α and β are constants; the reward function is used to balance the drone's mission completion and total energy consumption;

[0122] Step 4: Multi-agent entropy maximization algorithm

[0123] Step 4.1 The purpose of reinforcement learning is to learn the strategy p to maximize the objective of the training set; the entropy maximization algorithm improves the traditional objective function and adds an entropy factor on the basis of the expected reward to enhance the randomness of the agent's decision. The objective function can be described as:

[0124]

[0125] Among them, s t represents the state at time t, a t represents the action at time t, ρ π is the marginal condition of the state-action distribution, r(s t ,a t ) represents the state s at a certain moment t Next, perform action a t The reward obtained, H(π(a t |s t )) is the entropy of the strategy, α is the entropy temperature coefficient, which can be regarded as the weight coefficient of the entropy factor and the original reward function; Indicates that under a given strategy π, state s t and action a t According to the state-action distribution ρ π The expected value calculated.

[0126] The purpose of maximum entropy is to increase the randomness of the agent's decision-making. The concept of entropy is introduced by information entropy, which can be regarded as the probability distribution of the agent's behavior. The concept of entropy is introduced in the loss function to encourage the agent to explore, increase the randomness of its strategy, and avoid obtaining a local optimal solution. The expression of entropy is as follows:

[0127] H=log(π θ (a'|τ'))

[0128] π θ (a'|τ') means that given the information τ', the strategy π θ The probability of choosing action a'.

[0129] Step 4.2 Initialize the policy network π of each agent i θ , local value network And combined into a joint value function:

[0130]

[0131] where g ω It is a single-layer monotone mixture network.

[0132] The maximum value of the joint action value function is consistent with the maximum value of the single agent action value function

[0133]

[0134] Step 4.3 To evaluate the strategy, optimize the Critic network by minimizing the temporal difference loss. The difference loss is expressed as follows:

[0135]

[0136] in, is the current joint value function, y tot is the target value of the value function, y tot The calculated values ​​are as follows:

[0137]

[0138] in are the parameters of the target Critic, Actors and hybrid network respectively, and s is the generation bias of the input.

[0139] In step 4.4, during the policy update process, the factorized soft policy iteration method is used to optimize the policy network by maximizing the policy entropy and the joint value function:

[0140]

[0141] The joint action value function is decomposed into a linear combination of the critic value and entropy of the 10 agents, where ω i (s) is the weight and b(s) is the bias term.

[0142] Convert it into the expectation of the joint action-value function;

[0143]

[0144] Use substitution variables to simplify the expression using the new variable α:

[0145] α=ω i (s)α i

[0146] Introducing the joint strategy, simplifying the summation in the formula:

[0147]

[0148] The final form consists of two parts: the joint action value function and entropy:

[0149]

[0150] Step 5: Multi-area search task process:

[0151] Step 5.1 introduces the UAV energy model in step 2 to calculate the energy consumption and remaining stored energy of each UAV strategy selection (state switching);

[0152] Step 5.2 Use the priority planning strategy (PP) to calculate the list of target areas for each drone l t Priority, l t Expressed as

[0153] l t =[U1,U2...U 30 ]

[0154] Step 5.3 When each drone performs a task, its action decision is obtained by the multi-agent entropy maximization strategy algorithm in step 4. Compared with the completely independent multi-agent algorithm, the multi-agent entropy maximization algorithm introduces the system entropy term, increases the randomness of the system decision, avoids overload and falls into suboptimal solutions, and improves the overall performance of the agent system;

[0155] Step 5.4 In addition, considering that the drone is exhausted due to special conditions and cannot complete the task, the control system should return its target number to the task list and reallocate the available drones;

[0156]

[0157] Step 5.5: When the drone arrives at the sub-area, it will apply for a full coverage search inside the sub-area. After the drone completes the task, the drone's status will change from busy to idle, and the drone in idle status will be reassigned a new task.

[0158] The specific description above further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-agent UAV search task energy optimization method based on entropy maximization strategy, characterized by: The following steps are included: Step 1: Create a rasterized map of size K×K×K, determine and define the initial positions of n drones in the Cartesian coordinate system, and use m spatially distributed unconnected polygons as the target area U of the search task j , constraining the maximum search angle θ of a single UAV and the maximum effective search height d of the UAV max , assign the target area to n drones, and dispatch the drones to cover m polygonal areas to reach U j All coordinate points in space are targets that the drone is searching for; Step 2: Based on the equivalent area S of the drone body F , Inductive power factor κ p , blade drag coefficient σ e 、Blade resistance coefficient Ω e , rotation radius R e , air density ρ, rotor solidity R S 、UAV weight M UAV Establishing the UAV power propulsion model P F (V), and based on the UAV power driving model, the UAV energy consumption model E is established i (t); Step 3: Define the drone state space U i , based on the drone’s information S i , the observed environment information FOV, and the information of the target area T i Composition; define the action space A of the drone in the map; Define a reward function for ensuring the collision safety of the drone and optimizing the total energy consumption. Based on the reward function, the drone is guaranteed to have the collision safety and optimize the total energy consumption during the mission. The reward function includes the heading reward of the drone. Collision Penalty between Drones Penalty for drone crossing the boundary or touching the ground Energy consumption penalty for drones to complete missions Energy depletion penalty during drone missions Rewards for all drones reaching their targets Step 4: Add an entropy factor to the reinforcement learning objective function, build an entropy maximization strategy to make decisions on the drone's actions, and improve the drone's randomness and exploration capabilities when making decisions; Construct an entropy maximization strategy to make decisions on the actions of the drone. The specific implementation method is: Initialize the policy network π of each drone i θ , local value network And combined into a joint value function, the Critic network is optimized by minimizing the time difference loss, and the factorized soft policy iteration method is used to optimize the policy network by maximizing the policy entropy and the joint value function. The expectation of the joint value function is a linear combination of the value function and entropy of the i-agents. The joint strategy is introduced to obtain an objective function that combines the joint value and entropy terms and decomposes the complex joint value function into a form that can be optimized. During training, the entropy maximization policy network is optimized through iterative interaction, so that the entropy maximization policy gradually approaches the optimal; Step 5: Introduce the drone energy model in step 2, calculate the energy consumption and remaining stored energy of each drone strategy selection, use the priority planning strategy to plan the target area for each drone, and make decisions on the drone's actions based on the entropy maximization strategy in step 4. At the same time, consider the situation where the drone energy is exhausted and the drone cannot complete the task, reallocate the drone tasks, and achieve energy optimization of multi-agent drone search tasks.

2. The multi-agent UAV search task energy optimization method of the entropy maximization strategy according to claim 1 is characterized by: Step 1 is implemented as follows: n drones are distributed on the rasterized map, and the initial position of the drone in the Cartesian coordinate system is c i (i=1,2,...n), each drone moves at a constant speed. For a gridded map of size K×K×K, the state space of the drone is represented as S; S={place x }×{place y }×{place z } Among them, loc x ,loc y ,loc z ∈[0,K], representing c i The x, y, z coordinates of the position in the Cartesian coordinate system; Target area: There are m spatially distributed disconnected polygonal areas represented as: U j (j=1,2,...,m) Each region consists of a set of coordinate points close to the ground composition; The search angle of a single drone is θ, and the maximum effective search height of the drone is d max , when the coordinates of a certain drone are (x i ,y i ,z i ), the drone can effectively search for the coordinates (x, y, z) of the point: (z-z i ) 2 <d max 2 In the grid map, based on the UAV search angle constraint and the UAV search maximum height constraint, the target area is assigned to n UAVs, and the UAVs are dispatched to cover m polygonal areas to meet U j All coordinate points in space are effectively searched by the drone.

3. The multi-agent UAV search task energy optimization method of the entropy maximization strategy as claimed in claim 2 is characterized by: In step 2, Drone Power Propulsion Model P F (V) is expressed as: Among them, S F represents the equivalent area of ​​the fuselage, κ p represents the induced power factor, g represents the acceleration of gravity, σ e represents the blade drag coefficient, Ω e represents the blade drag coefficient, R e represents the rotation radius, ρ represents the air density, R S Indicates rotor solidity, M UAV Indicates the weight of the drone. v0 represents the average rotor speed during hovering, d0 is the fuselage drag ratio, s is the rotor solidity, V represents the flight speed, and A represents the propeller area. The drone is in the time span T H Energy consumption E i (t) is expressed as: E i (t)=P i T (t)T H Since each drone maintains a constant flying speed, there is a positive correlation between the energy consumption of a single drone and the flight distance.

4. The multi-agent UAV search task energy optimization method of the entropy maximization strategy as claimed in claim 3 is characterized by: Step 3 is implemented as follows: The state space is the information S of the drone i , the observed environment information FOV, and the information of the target area T i A collection of; U i =(S i ,FOV i ,E S ,T i ) in, id represents the serial number of the drone, pos represents the position of the drone in the Cartesian coordinate system, γ represents the deflection angle of the drone, and C∈{idle,busy,depleted} represents the current state of the drone, which is idle, working, and energy depleted. E S Indicates the remaining energy of the drone, expressed as: in, represents the initial energy of UAV i; t is the continuous working time; T i The unit vector pointing from the drone to the next area The Euclidean distance l from the drone to the target is composed of: At time t, through the action a of drone i i t Information about the drone i t , calculate the information S of the drone at the next moment i t+1 ; By updating S i Get the state space at each moment; The discrete action space of the drone is represented as A={turn left ,turn right ,rise,decline,maintain} turn left Indicates that the drone turns left, turn right The drone deflects to the right, rise means the drone is rising, decline means the drone is descending, and maintain means the drone maintains the current horizontal direction and moves in a straight line; L means left, the right deflection angle is the same; the rise and fall angles are the same; The reward function of the drone includes the following parts: Expressed as Collision Penalty between Drones Defined as parameter p1, the penalty for the drone crossing the boundary or touching the ground Defined as parameter p2, the energy consumption penalty of the drone under the condition of completing the task Expressed as -β|E max -E s |, Energy depletion penalty during drone missions Defined as parameter p3, the reward for all drones reaching the target Defined as parameter p4; in, Representation vector Angle in Cartesian coordinate system; γ t Represents the angle of the vector from the current position of the drone to the target point; α and β are constants; based on the reward function, the drone is guaranteed to have collision safety and optimize total energy consumption during the mission.

5. The multi-agent UAV search task energy optimization method of the entropy maximization strategy according to claim 4 is characterized by: The specific implementation method of step 4 is: The reward function of the entropy maximization method adds an entropy factor to the reward function in step 3 to enhance the randomness of the decision-making of the drone's actions, so that the drone can explore more for the next action, increase the randomness of its strategy, and avoid obtaining a local optimal solution. The objective function after adding the entropy factor is: Among them, s t represents the state at time t, a t represents the action at time t, ρ π is the marginal condition of the state-action distribution, r(s t ,a t ) represents the state s at a certain moment t Next, perform action a t The reward obtained, H(π(a t |s t )) is the entropy of the strategy, α is the entropy temperature coefficient, which can be equivalent to the weight coefficient of the entropy factor and the original reward function; Indicates that under a given strategy π, state s t and action a t According to the state-action distribution ρ π The expected value calculated; Among them, entropy is used to characterize the probability distribution of each drone action, and its expression is as follows: H=log(π θ (a'|t')) π θ (a'|τ') means that given the information τ', the strategy π θ The probability of selecting action a', where action a' is contained in the action space A in step 3; Initialize the policy network π of each drone i θ , local value network And combined into a joint value function: Among them, g ω It is a single-layer monotone mixing network; The maximum value of the joint value function is consistent with the maximum value function of a single drone Sampling and interaction: The drone interacts with the environment based on the current strategy π θ Select action a t ; After executing the action, the environment feedbacks the new state s t+1 and instant rewards t ;Store the interaction data in the experience pool for subsequent training, wherein the interaction data includes state, action, reward, and new state; Evaluation strategy, optimizes the Critic network by minimizing the temporal difference loss. The difference loss is expressed as follows: in, is the current joint value function, y tot is the target value of the value function, y tot The calculated values ​​are as follows: in are the parameters of the target Critic, Actors and hybrid network respectively, and s is the input generation bias; Policy update uses a factorized soft policy iteration method to optimize the policy network by maximizing the policy entropy and the joint value function: The expectation of the joint value function is a linear combination of the value functions and entropy of the i agents, where ω i (s) is the weight, b(s) is the bias Introducing a joint strategy The entropy terms of each agent are summarized into an overall entropy expression for further calculation: Obtain an objective function that combines the joint value and entropy terms and decomposes the complex joint value function into an optimizable form, so that during training, the policy network is optimized by maximizing this objective function; Target network update: Regularly perform soft updates on the target network to enhance the stability of training: φ'←τφ+(1-τ)φ' Where, τ is the soft update coefficient; Repeat the above steps of evaluating strategy, updating strategy, and updating target network. Through multiple iterative interactive optimizations, the entropy maximization strategy gradually approaches the optimal.

6. The multi-agent UAV search task energy optimization method of the entropy maximization strategy according to claim 5 is characterized by: Step 5 is implemented as follows: Use the priority planning strategy to calculate the target area list l for each drone t Priority, l t Expressed as the t =[U1,U2...U n ] When each drone performs a task, its action decision is obtained by the entropy maximization strategy of the four multi-agents in step 4. Compared with the completely independent multi-agent algorithm, the multi-agent entropy maximization strategy introduces the system entropy term to increase the randomness of the drone's decision, avoid overload and fall into suboptimal solutions, and improve the overall performance of the agent system. Considering the situation where the drone is unable to complete the task due to exhaustion of energy, the energy optimization of the multi-agent drone search task is realized. The control system returns its target number to the task list and reallocates the available drones; When the drone arrives at the sub-area, it will apply for a full coverage search inside the sub-area. After the drone completes the task, the drone's status will change from busy to idle, and the drone in the idle state will be rescheduled for a new task.

Images