A heterogeneous unmanned aerial vehicle regional coverage method, system and device

Abstract

The application discloses a heterogeneous unmanned aerial vehicle area coverage method, system and device, relates to the multi-agent system planning and control technical field, and the method comprises the following steps: collecting the interaction data of each unmanned aerial vehicle and the environment; training a multi-agent reinforcement learning algorithm based on the interaction data of each unmanned aerial vehicle and the environment, and constructing a heterogeneous unmanned aerial vehicle coverage network model; inputting the position information of each unmanned aerial vehicle into the heterogeneous unmanned aerial vehicle coverage network model, and outputting the expected attitude of each unmanned aerial vehicle; and completing the coverage path planning of the heterogeneous unmanned aerial vehicle according to the expected attitude of each unmanned aerial vehicle. In the method, each unmanned aerial vehicle can autonomously calculate the current optimal coverage path planning based on the current observation of the environment and the information interaction with neighbor unmanned aerial vehicles, so that the heterogeneous unmanned aerial vehicle can obtain the optimal control strategy, and real-time coverage planning and control are realized.

Description

Technical Field

[0001] This invention relates to the field of multi-agent system planning and control technology, specifically to a heterogeneous unmanned aerial vehicle (UAV) area coverage method, system, and device. Background Technology

[0002] Comprehensive coverage and monitoring of a target area is one of the important tasks in the application of multi-UAV systems. When UAVs perform coverage tasks, this is specifically achieved by multiple UAVs jointly covering a given area with their field of vision. Each UAV sends the information it collects to a central ground controller, which can then perform actions such as local map building or path planning based on this information.

[0003] In multi-UAV systems, increasing the number of UAVs does not necessarily improve system performance due to the communication burden between UAVs and the computational burden on the central processor. To ensure that the multi-UAV system can fully collect information about the target area while reducing the system load, precise online planning and control of the movement and behavior of the UAV swarm are required to achieve optimal coverage of the target area.

[0004] Traditional UAV coverage control algorithms primarily employ the Thiessen polygon method for region partitioning. This method assumes all UAVs possess identical dynamic characteristics and mission capabilities, thus simplifying UAVs into isomorphic objects for coverage area optimization. In practical applications, this assumption ignores the performance and functional differences between UAVs, such as varying flight speeds, payload capacities, and endurance, leading to significant reductions in coverage efficiency and accuracy. Especially when facing dynamically changing environments or mission requirements, isomorphic algorithms exhibit extremely low flexibility and adaptability, failing to effectively adjust the specific roles and responsibilities of each UAV.

[0005] In recent years, a coverage control strategy based on neural networks for multi-UAV systems has been proposed and has achieved good results. However, such methods have very complex neural networks and require a long training time. In addition, most coverage control methods only consider the planning level, that is, they assume that each UAV can accurately execute the expected action, which is not realistic. Therefore, the current coverage control methods for multi-UAV systems still have the following problems: (1) Most coverage methods are a centralized control method, which requires a central control node to control each UAV in real time, and lacks timeliness and scalability; (2) They do not consider the heterogeneity of multi-UAV systems, that is, the structure and detection capabilities of each UAV in a multi-UAV system are different; this will reduce the coverage efficiency of multi-UAV systems; (3) Multi-UAV coverage control methods based on neural networks have complex network structures, long training times, and lack the ability to be deployed quickly; (4) They do not consider the dynamic constraints of actual UAVs, that is, they cannot guarantee that the UAV will correctly execute the expected action. In addition, traditional coverage algorithms consider UAVs as homogeneous objects, but in reality, UAVs have different dynamic models and tasks in cluster tasks, so it is difficult to transfer them to real-world scenarios. Summary of the Invention

[0006] To address the shortcomings of existing technologies that fail to consider the heterogeneity of multiple UAV systems when performing coverage control over a region, treating UAVs as homogeneous objects and making them difficult to adapt to real-world scenarios, this invention proposes a heterogeneous UAV regional coverage method, system, and device. By utilizing reinforcement learning to complete coverage path planning for heterogeneous UAVs, it achieves real-time coverage planning and control, thereby solving the problems existing in the prior art.

[0007] A method for heterogeneous unmanned aerial vehicle (UAV) area coverage includes the following steps:

[0008] Collect historical interaction data between multiple UAVs and the target area; the interaction data includes the current observation data of each UAV and the observation data of neighboring UAVs;

[0009] A heterogeneous drone coverage network model is constructed by training the ACTOR-CRITIC reinforcement learning method using historical interaction data between multiple drones and a target area. Specifically, this involves: obtaining the target area's reward feedback, the drone's current observation value, and the new observation value obtained after executing an action, based on each drone's current observation data and the observation data of its neighboring drones. These new observation values ​​are then grouped into state tuples and stored in the experience replay pool of the CRITIC state value function network. For any given state tuple, a message passing mechanism based on an attention network is used to extract the feature values ​​of each drone during multiple rounds of message passing with its neighboring drones. The CRITIC state value function network for each drone is generated based on these feature values. Finally, the parameters of the CRITIC state value function network and the ACTOR policy network are updated using stochastic gradient descent to construct the heterogeneous drone coverage network model.

[0010] Real-time interaction data between multiple drones and the target area is input into a heterogeneous drone coverage network model, which outputs the desired attitude of each drone. Each drone achieves coverage of the target area by tracking its own desired attitude.

[0011] Furthermore, the process of acquiring the reward feedback of the target area, the current observation value of the UAV, and the new observation value obtained after performing the action, and then saving the new observation value as a state tuple to the experience replay pool of the CRITIC state value function network, specifically includes the following steps:

[0012] Define the set of m vertex coordinates of the target region as [e1,...,e...]. m The coordinates of drone i are p. i The closest N to drone i i The coordinate set of the drones is Each drone i obtains the current environment s t Observations for:

[0013]

[0014] Based on the current policy network π θ Execute action Where θ represents the policy network parameters;

[0015] Each drone i is based on its current state With observation Execute action The drone's status is updated after interacting with the environment. At the same time, receive reward feedback from the environment. and new observations The environment refers to the target area;

[0016] Environmental reward feedback and the actions performed by the drone Observations form state tuples Save it to the experience replay pool for value function network maintenance.

[0017] Furthermore, the step of selecting any state tuple and using an attention network-based message passing mechanism to extract the feature values ​​of each drone during multiple rounds of message passing with neighboring drones includes the following steps:

[0018] Select d state tuples from the current experience replay pool. constitute

[0019] Based on the state tuple of the i-th drone in D Get the state information p of the i-th drone i And according to the group code g of the drone i After obtaining the coded UAV group information, the position vector p' i =concat[p i ,g i ], traversing N drones, we obtain [p'1, p'2, ..., p' N ];

[0020] Define the message sequence received by drone i from neighboring drones as follows: k refers to the message transmitted in the k-th round of a message passing process, for each drone i: The message received by drone i in the initial stage is

[0021] The drone exchanges messages with its neighboring drones in k rounds, with each round based on the messages processed in the previous round. Furthermore, based on an attention network mechanism, the heterogeneous information from the neighbors is encoded using attention units to obtain Q, K, and V, which are represented as follows:

[0022]

[0023] Among them W Q W K W V Given the linear transformation matrix for different attention units, the message transmitted by the drone in the (k+1)th round is:

[0024]

[0025] Where d k To query the dimension of vector Q, f a Represents an attention network;

[0026] The eigenvalues ​​are obtained after k rounds of propagation. The message received by the i-th drone after the k-th round of transmission. Represented as:

[0027]

[0028] Furthermore, the CRITIC function, which calculates the state value function for each UAV based on its eigenvalues, is used to determine the state of UAV i in state s. i The network calculation formula for the state value function at time is:

[0029]

[0030] Among them, f w This represents a fully connected network where all drones share a single state-value function network V. φ .

[0031] Furthermore, it also includes using a reward reconstruction mechanism to calculate the reward for each drone i and the global reward r. glob Local reward r loc The final reward r for training the state-value function network parameters is given by the drone i. i train Specifically, it is expressed as:

[0032]

[0033] Where, k1,k loc ,k glob These are the reward and the local reward, respectively. loc Global reward r glob The corresponding reward weight, Let be the flight energy consumption coefficient of the i-th UAV.

[0034] Furthermore, the method of updating the CRITIC parameters of the state value function network and the ACTOR parameters of the policy network using stochastic gradient descent specifically includes:

[0035] The network parameters φ of the state value function are updated using stochastic gradient descent, denoted as:

[0036]

[0037] in, For samples from k iterations, the objective value of the function is... This is the cumulative discounted return, estimated using the Monte Carlo cumulative return method, expressed as:

[0038]

[0039] Where γ is the discount rate;

[0040] The network parameters θ updated using stochastic gradient descent are expressed as follows:

[0041]

[0042] Where, π θ (a t |s t ) is the policy function. It is the state-action value function, V φ (s t ) is the value function, and ε∈[0,1] is a user-defined hyperparameter.

[0043] This invention also proposes a control system for heterogeneous unmanned aerial vehicle (UAV) area coverage, comprising:

[0044] The data acquisition module is used to collect historical interaction data between multiple UAVs and the target area; the interaction data includes the current observation data of each UAV and the observation data of neighboring UAVs.

[0045] The model building module is used to train the ACTO R-CRITIC reinforcement learning method using historical interaction data between multiple UAVs and the target area, constructing a heterogeneous UAV coverage network model. Specifically, it includes: obtaining the reward feedback of the target area, the current observation value of the UAV, and the new observation value obtained after executing an action, based on the current observation data of each UAV and the observation data of its neighboring UAVs. These new observation values ​​are then grouped into state tuples and saved to the experience replay pool of the CRITIC state value function network. Any state tuple is selected, and a message passing mechanism based on an attention network is used to extract the feature values ​​of each UAV during multiple rounds of message passing with its neighboring UAVs. The CRITIC state value function network for each UAV is generated based on these feature values. Finally, the parameters of the CRITIC state value function network and the ACTOR policy network are updated using stochastic gradient descent to construct the heterogeneous UAV coverage network model.

[0046] The coverage module is used to input real-time interaction data between multiple UAVs and the target area into the heterogeneous UAV coverage network model and output the desired attitude of each UAV; each UAV achieves coverage of the target area by tracking its own desired attitude.

[0047] The present invention also proposes a control computer device for heterogeneous UAV area coverage, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of controlling the heterogeneous UAV area coverage.

[0048] The present invention also proposes a readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, are used to perform the steps of the heterogeneous UAV area coverage method.

[0049] This invention provides a method, system, and apparatus for heterogeneous unmanned aerial vehicle (UAV) area coverage, which has the following beneficial effects:

[0050] This invention constructs a coverage network model for heterogeneous drones using multi-agent reinforcement learning algorithms, based on the observation data of each drone and those of neighboring drones. When covering a target area, no central control node or human control is required. Each drone can autonomously calculate the optimal coverage path based on its own observation of the environment and information exchange with neighboring drones, thereby completing the coverage path planning for heterogeneous drones. This allows the heterogeneous drones to obtain the optimal control strategy and achieve real-time coverage planning and control. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the training process of the heterogeneous UAV overlay network model in an embodiment of the present invention.

[0052] Figure 2 This is a schematic diagram of the state value function network structure using the attention mechanism in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the heterogeneous UAV strategy network structure in an embodiment of the present invention;

[0054] Figure 4 This is a block diagram of a heterogeneous UAV coverage system based on multi-agent reinforcement learning in an embodiment of the present invention;

[0055] Figure 5 This is a hardware architecture diagram of a heterogeneous UAV coverage system based on multi-agent reinforcement learning in an embodiment of the present invention;

[0056] Figure 6 This is a flowchart of the method for heterogeneous drone coverage in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0058] This invention proposes a method for heterogeneous unmanned aerial vehicle (UAV) coverage, such as... Figure 6As shown, this invention utilizes reinforcement learning to perform coverage path planning for heterogeneous UAVs and designs an attitude controller to track the attitude output by the neural network. Specifically, it includes an offline training phase for the deep reinforcement learning network and an online usage phase. In offline training, the invention employs the ACTOR-CRITIC method to train the network, such as... Figure 1 As shown in the diagram. The ACTOR network generates the expected policy for each drone in the next moment, while the CRITIC network evaluates the drone's reward after performing its current action. Furthermore, the CRITIC network's specific structure consists of attention units and a fully connected MLP layer, as shown in the diagram. Figure 2 As shown, the purpose of this design is to enable the network to extract effective information from a large amount of data, thereby reducing the instability caused by constantly changing learning objectives. The ACTOR network structure is as follows: Figure 3 As shown, it consists of two layers of fully connected neural networks and one layer of nonlinear activation unit ReLU; specifically, it includes the following steps:

[0059] Offline training phase steps:

[0060] S1. Based on the heterogeneous characteristics of UAVs, the UAVs are grouped and coded. The group code for UAV i is g. i Agents in the same group have the same encoding parameters; initialize ACTOR and CRITI C networks with network parameters initialized to θ, θ', φ, φ' respectively; initialize a message passing mechanism based on an attention network for each UAV, the purpose of which is to establish a communication mechanism between UAVs so that the UAVs can obtain more comprehensive information.

[0061] S2. Collect data on the interaction between the drone and the environment. Define the vertex coordinates of the area to be covered as e. i The coordinates of drone i are p i Each drone i obtains its current state s. t Observations for:

[0062]

[0063] Each drone i is determined by the observations at the current moment. Get the state information p of the i-th drone i And according to the group code g of the drone i Obtain new state information p' i =concat[p i ,g i ], traversing N agents to obtain [p1', p'2, ..., p' N Use it as the ACTOR network input, and then obtain the network output. This is the expected action of the drone in the next moment. The drone completes the action. Then, update to the new state s t →s t+1 The effectiveness of actions is evaluated by the CRITIC network, which then provides reward feedback. And obtain the observation at the next time step. Reward feedback and the actions performed by the drone Observations form state tuples Save it to the experience replay pool for value function network maintenance.

[0064] S3. Train the ACTOR and CRITIC networks. After performing step S2 a finite number of times, retrieve d state tuples from the current experience replay pool. constitute To obtain more comprehensive global information and avoid the impact of local observation problems on algorithm convergence, the UAV conducts multiple rounds of message passing with neighboring UAVs. An attention-based message passing mechanism is used to extract features, obtaining feature values ​​(f1, f2, ... f). N Each round of message passing is based on the messages processed in the previous round, and the specific steps are as follows:

[0065] S3.1, the state tuple of the i-th UAV in D Get the state information p of the i-th drone i And according to the group code g of the drone i Obtain new state information p i '=concat[p i ,g i ], traversing N agents yields [p'1, p'2, ..., p' N ].

[0066] S3.2, Define the message sequence received by agent i from neighboring agents as follows: Where k refers to the message passed in the k-th round of a message passing process. For each drone i, initialization... The message received by drone i in the initial stage is

[0067] The agent exchanges messages with its neighboring agents for k rounds. Each round of message exchange is based on the messages processed in the previous round. According to the basic principle of attention units, the UAV encodes the neighbor information through attention units to obtain Q, K, and V.

[0068]

[0069] Where h(*) represents the nonlinear encoding of the neural network, WQ W K W V Given the linear transformation matrix for different attention units, the message transmitted by the drone in the (k+1)th round is:

[0070]

[0071] Where d k The dimension of the query vector Q.

[0072] The eigenvalues ​​are obtained after k rounds of propagation. According to the above formula, the message obtained by the i-th agent after the k-th round of propagation is... This can be expanded as follows:

[0073]

[0074] S4. From the eigenvalues ​​(f1, f2, ... f N ) Calculate the state value function v of the data unit φ (f1),v φ (f2),...v φ (N), all agents share a CRITIC network V φ For agent i, which is in state s i The value function is calculated as follows:

[0075]

[0076] Among them, f w Represents a fully connected network, f a The network represents the attention network, and all drones share a single value function network V. φ .

[0077] S5. Update ACTOR network parameters φ.

[0078] S5.1 Design a reward function based on a reward reconstruction mechanism.

[0079] Given a target region Q∈R 2 Let q be the number of points within the target region, and s be the set of points within the target region. obj The coverage area of ​​drone i is defined as follows:

[0080]

[0081] Scalar function φ(q):R 2 →R + This is a mapping from the coordinates of each point within the region to (0,1), used to represent the importance of each point. The controller's position is p. μ The rate of change of weights φ(q) is defined as:

[0082]

[0083] in, In a heterogeneous drone swarm, different drones have different energy loss coefficients.

[0084] In the design of reward functions for heterogeneous multi-UAV coverage control, the reward function differs slightly from that of homogeneous multi-agent algorithms due to the differences between agents. Agents have different speeds and coverage ranges; in this case, using a global reward to train the model will lead to lazy agents. Therefore, we consider introducing a reward based on local observations for each agent in addition to the global reward to address the reputation allocation problem of agents regarding the global reward. First, the reward for each agent i is calculated using the following formula: the global reward r... glob and local reward r loc The final reward r for agent i used to train the value function network parameters. i train In the formula: As defined above, φ(q) is used.

[0085]

[0086] r i train =k glob ·r glob +k loc ·r loc

[0087] S5.2 Update the ACTOR network parameters φ using stochastic gradient descent.

[0088]

[0089] definition Given samples from k iterations, γ is the discount rate, and the objective value of the function is... This is the cumulative discounted return, estimated using the Monte Carlo cumulative return method, as shown below:

[0090]

[0091] S6. Update the policy network parameters θ using stochastic gradient descent as follows:

[0092]

[0093] S7. Keep the target network parameters consistent with the original network parameters: θ′=θ, φ′=φ.

[0094] S8. Repeat S3-S7 until the network training is complete.

[0095] The online usage phase includes the following steps:

[0096] S1, Each UAV i acquires the observations at the current moment.

[0097] S2. Each UAV exchanges observations with neighboring UAVs based on an information exchange mechanism. Assume the state information p of the i-th UAV... i And according to the group code g of the drone i Obtain new state information p' i =concat[p i ,g i ], traversing N agents yields [p'1, p'2, ..., p' N ].

[0098] S3, Drone i will [p'1,p'2,...,p' N Use it as the ACTOR network input, and then obtain the network output. As its expected action in the next moment.

[0099] S4. Perform tracking of the desired trajectory within each drone. Figure 4 This is a schematic diagram of the tracking control algorithm architecture. The velocity information output by the policy network yields the expected pose of the UAV at the next time step. The UAV's attitude controller, based on the current pose of the UAV and the expected pose output by the neural network, calculates the control torque and outputs it as a command to the UAV model. The UAV outputs the motor speed based on the relationship between the motor and the control torque. The framework is as follows: Figure 3 As shown. The output of the policy network is the speed of the UAV. First, the UAV mass and the policy output of the network are simplified into a first-order model, as shown below:

[0100]

[0101] Where s i This represents the position of the current UAV i and its neighboring UAVs relative to the global coordinate system. x, v, m, R, Ω, J, M represent the UAV's position, velocity, mass, attitude matrix, rotational angular velocity, moment of inertia, and torque input, respectively, for the following UAV flight dynamics model:

[0102]

[0103] The differential equation for attitude error can be obtained as follows, where subscripts e and d represent the error value and expected value of each physical quantity, respectively.

[0104]

[0105] The desired torque input is:

[0106]

[0107] The attitude error matrix can be derived from the quaternion error vector (q). e0 ,q e1 ,q e2 ,q e3 The values ​​of q and error matrix are obtained by mutual conversion, where q ei Let represent the error in the i-th dimension. The transformation relationship is shown in the equation:

[0108]

[0109] Each drone uses an external positioning system, such as GPS, to obtain its own position, providing real-time coordinates for each drone. The systems communicate with each other via ROS. Figure 5 As shown. The host computer subscribes to coordinate topics published by VICON to obtain the real-time positions of the drone swarm. The model is deployed on a host computer. After receiving the data, the host computer converts it into standard input for a neural network, then transmits the data to the neural network to obtain action output. Based on the obtained action and the current position of the drone, the waypoint of the drone at the next moment is obtained, and then the waypoint is published to each drone through the PA wireless sensor that comes with CRAZYFLIE. The attitude tracking controller of each drone attempts to track the target attitude to complete the coverage of the target area.

[0110] Based on the same inventive concept, this invention also proposes a control system for heterogeneous unmanned aerial vehicle (UAV) area coverage, comprising:

[0111] The data acquisition module is used to collect historical interaction data between multiple drones and the target area; the interaction data includes the current observation data of each drone and the observation data of neighboring drones.

[0112] The model building module is used to train the ACTO R-CRITIC reinforcement learning method using historical interaction data between multiple UAVs and the target area, constructing a heterogeneous UAV coverage network model. Specifically, it includes: obtaining the reward feedback from the target area, the current observation value of the UAV, and the new observation value obtained after executing an action, based on the current observation data of each UAV and the observation data of its neighboring UAVs. These new observation values ​​are then grouped into state tuples and saved to the experience replay pool of the CRITIC state value function network. Any state tuple is selected, and a message passing mechanism based on an attention network is used to extract the feature values ​​of each UAV during multiple rounds of message passing with its neighboring UAVs. The CRITIC state value function network for each UAV is generated based on these feature values. Finally, the parameters of the CRITIC state value function network and the ACTOR policy network are updated using stochastic gradient descent to construct the heterogeneous UAV coverage network model.

[0113] The coverage module is used to input real-time interaction data between multiple UAVs and the target area into the heterogeneous UAV coverage network model and output the desired attitude of each UAV; each UAV achieves coverage of the target area by tracking its own desired attitude.

[0114] The present invention also proposes a control computer device for heterogeneous UAV area coverage, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the control steps for heterogeneous UAV area coverage.

[0115] The present invention also proposes a readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, are used to perform the steps of the heterogeneous UAV area coverage method.

[0116] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for regional coverage by heterogeneous unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: Collect historical interaction data between multiple UAVs and the target area; the interaction data includes the current observation data of each UAV and the observation data of neighboring UAVs; A heterogeneous drone coverage network model is constructed by training the ACTOR-CRITIC reinforcement learning method using historical interaction data between multiple drones and a target area. Specifically, this includes: obtaining the reward feedback from the target area, the drone's current observation value, and the new observation value obtained after executing an action, based on each drone's current observation data and the observation data of its neighboring drones. These new observation values ​​are then grouped into state tuples and stored in the experience replay pool of the CRITIC state value function network. Any state tuple is selected, and a message passing mechanism based on an attention network is used to extract the feature values ​​of each drone during multiple rounds of message passing with neighboring drones. The CRITIC state value function network for each drone is generated based on these feature values. The parameters of the CRITIC state value function network and the ACTOR policy network are updated using stochastic gradient descent to construct the heterogeneous drone coverage network model. The step of selecting any state tuple and using the message passing mechanism based on an attention network to extract the feature values ​​of each drone during multiple rounds of message passing with neighboring drones includes the following steps: selecting from the current experience replay pool... state tuple data ,constitute ,in For each drone Get the current environment The observed values, For the current policy network The action performed Rewards and feedback for the environment; based on The Middle The state tuple of a drone Get the Status information of each drone And according to the group code of the drone. Position vector after obtaining coded UAV group information traversal One drone received Define drones The message sequence received from the neighbor's drone is , This refers to the first message in a single message transmission. The message is passed in rounds, for each drone : For drones in the initial stage The message received is ; Drones interacting with neighboring drones The message passing process is repeated in rounds, with each round based on the messages processed in the previous round; and the heterogeneous neighbor information is encoded through attention units according to the attention network mechanism to obtain... It is represented as: ； in For different linear transformation matrices of the attention units, the UAV in the th... The message passed in the round is: ;in For query vector Dimensions Represents the attention network; after Eigenvalues ​​are obtained after round propagation Among them, the first The drone in the first Message received after round passing Represented as: ; Real-time interaction data between multiple drones and the target area is input into a heterogeneous drone coverage network model, which outputs the desired attitude of each drone. Each drone achieves coverage of the target area by tracking its own desired attitude.

2. The heterogeneous UAV area coverage method according to claim 1, characterized in that, The process of acquiring reward feedback from the target area, the current observations of the UAV, and new observations obtained after executing actions, and then saving these new observations as a state tuple to the experience replay pool of the CRITIC state value function network, specifically includes the following steps: Define target region The set consisting of the coordinates of each vertex is drones The coordinates are Distance from drone Recent The coordinate set of the drones is Each drone Get the current environment Observations for: ； Based on the current policy network Execute action ,in For policy network parameters; Each drone Based on the current state With observation Execute action After interacting with the environment, the drone's status is updated to... At the same time, it provides environmental rewards and feedback. and new observations The environment refers to the target area; Environmental reward feedback and the actions performed by the drone The observed values ​​form the state tuple. Save it to the experience replay pool for value function network maintenance.

3. The heterogeneous UAV area coverage method according to claim 2, characterized in that, The CRITIC state value function for each UAV is calculated based on its eigenvalues. In state The network calculation formula for the state value function at time is: ； in, This represents a fully connected network where all drones share a single state-value function network. .

4. The heterogeneous UAV area coverage method according to claim 3, characterized in that, It also includes using a reward-based refactoring mechanism to calculate the performance of each drone. Rewards, Global Rewards Local rewards and drones The final reward used to train the network parameters of the state-value function Specifically, it is expressed as: ； in, Rewards and partial rewards are respectively. Global Rewards The corresponding reward weight, For the first The flight energy consumption coefficient of a drone.

5. A heterogeneous unmanned aerial vehicle (UAV) area coverage method according to claim 3, characterized in that, The method of updating the CRITIC parameters of the state value function network and the ACTOR parameters of the policy network using stochastic gradient descent specifically includes: The network parameters of the state value function are updated using stochastic gradient descent. , represented as ； in, for The sample of the next iteration, the target value of the function. This is the cumulative discounted return, estimated using the Monte Carlo cumulative return method, expressed as: ； in, Discount rate; Stochastic gradient descent is used to update the policy network parameters. Represented as: ； in, For the policy function, It is the state-action value function. For value function, For custom hyperparameters.

6. A heterogeneous unmanned aerial vehicle (UAV) area coverage system, characterized in that, include: The data acquisition module is used to collect historical interaction data between multiple UAVs and the target area; the interaction data includes the current observation data of each UAV and the observation data of neighboring UAVs. The model building module is used to train the ACTOR-CRITIC reinforcement learning method using historical interaction data between multiple UAVs and a target area, constructing a heterogeneous UAV coverage network model. Specifically, it includes: obtaining the reward feedback of the target area, the current observation value of the UAV, and the new observation value obtained after executing an action, based on the current observation data of each UAV and the observation data of its neighboring UAVs; storing these new observation values ​​as state tuples in the experience replay pool of the CRITIC state value function network; selecting any state tuple and using a message passing mechanism based on an attention network to extract the feature values ​​of each UAV during multiple rounds of message passing with neighboring UAVs; generating the CRITIC state value function network for each UAV based on the feature values; and updating the parameters of the CRITIC state value function network and the ACTOR policy network using stochastic gradient descent to construct the heterogeneous UAV coverage network model. The step of selecting any state tuple and using a message passing mechanism based on an attention network to extract the feature values ​​of each UAV during multiple rounds of message passing with neighboring UAVs includes the following steps: selecting from the current experience replay pool... state tuple data ,constitute ,in For each drone Get the current environment The observed values, For the current policy network The action performed Rewards and feedback for the environment; based on The Middle The state tuple of a drone Get the Status information of each drone And according to the group code of the drone. Position vector after obtaining coded UAV group information traversal One drone received Define drones The message sequence received from the neighbor's drone is , This refers to the first message in a single message transmission. The message is passed in rounds, for each drone : For drones in the initial stage The message received is ; Drones interacting with neighboring drones The message passing process is repeated in rounds, with each round based on the messages processed in the previous round; and the heterogeneous neighbor information is encoded through attention units according to the attention network mechanism to obtain... It is represented as: ； in For different linear transformation matrices of the attention units, the UAV in the th... The message passed in the round is: ;in For query vector Dimensions Represents the attention network; after Eigenvalues ​​are obtained after round propagation Among them, the first The drone in the first Message received after round passing Represented as: ; The coverage module is used to input real-time interaction data between multiple UAVs and the target area into the heterogeneous UAV coverage network model and output the desired attitude of each UAV; each UAV achieves coverage of the target area by tracking its own desired attitude.

7. A heterogeneous unmanned aerial vehicle (UAV) area coverage computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the heterogeneous unmanned aerial vehicle area coverage method according to any one of claims 1-5.

8. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which includes program instructions that, when executed by a processor, are used to perform the steps of the heterogeneous unmanned aerial vehicle area coverage method according to any one of claims 1-5.

Images