A collaborative search method between drones and unmanned ships based on reinforcement learning

Abstract

The present invention discloses a method for collaborative search of unmanned aerial vehicles and unmanned ships based on reinforcement learning, which belongs to the field of collaborative search technology in marine environments, and includes the following steps: step 1, deploying multiple unmanned aerial vehicles and unmanned ships in a mission area; step 2, discretizing the mission area into a grid, and using a probability graph model to record the probability of target existence in each cell; step 3, when the energy of the unmanned aerial vehicle is insufficient, a charging request is sent to the unmanned ship, and the unmanned ship plans a path according to the energy of the battery of the unmanned aerial vehicle and the probability of target existence; step 4, the unmanned ship optimizes its own motion trajectory using a collaborative optimization scheduling algorithm, provides charging services for the unmanned aerial vehicle and performs target search; if no sea surface target is found, step 2 is re-executed; if the energy of the unmanned aerial vehicle is insufficient to complete the task, steps 3-4 are repeated until the target position is identified by the target probability. The present invention is suitable for large-scale collaborative search tasks of unmanned aerial vehicles and unmanned ships under marine environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of collaborative search in marine environments, and specifically relates to a reinforcement learning-based collaborative search method for unmanned aerial vehicles and unmanned ship captains. Background Art

[0002] Unmanned systems such as unmanned aerial vehicles (UAVs) and unmanned surface vessels (USVs) have been used in various maritime activities, including environmental monitoring, data transmission, cargo transportation, and emergency rescue. However, current autonomous maritime search systems still face problems such as low autonomy and poor search efficiency. In addition, due to the low position of the sensor equipment (such as cameras and radars) installed on USVs, their sensing range is limited. To address this problem, heterogeneous collaborative search and rescue systems of UAVs and unmanned vessels have received more attention, combining the long endurance of unmanned vessels with the wide field of view of UAVs, realizing their complementary advantages and improving search and rescue efficiency. However, UAVs perform complex and persistent operations such as video streaming and image processing, which may consume a lot of battery energy during the mission.

[0003] Currently, in order to solve the energy limitation problem of UAVs, unmanned ground vehicles (UGVs) are usually used as energy carriers, following pre-planned routes to rendezvous with UAVs regularly for charging. However, due to the unpredictable and diverse marine environment, this fixed-route search method is not suitable for complex marine search scenarios. In the existing literature, few studies have attempted to design marine mobile wireless charging systems to establish a viable on-demand solution to maintain large-scale continuous UAV operations.

[0004] Unmanned ships have the advantages of high efficiency, flexibility, and long cruising time. In existing solutions, drones recharge by landing on unmanned ships, which will cause mission interruptions. With the advancement of wireless power transmission (WPT) technology in recent years, a non-contact, fully automatic wireless charging solution has been provided for drones, allowing drones to charge during mission execution. Therefore, unmanned ships can provide continuous charging services for drones by being equipped with wireless charging facilities as mobile charging stations at sea, meeting the requirements of long-endurance missions. In the dynamic heterogeneous collaborative search scenario of drones and unmanned ships, how to flexibly and effectively solve the unmanned endurance problem is a key issue that needs to be solved urgently. Summary of the invention

[0005] In order to solve the problem of limited UAV endurance in maritime search scenarios, the present invention proposes a UAV and unmanned ship endurance collaborative search method based on reinforcement learning. A multi-agent deep reinforcement learning algorithm is used to adjust the moving direction of the unmanned ship and optimize the motion trajectory to provide charging services for the UAV and perform target search, and to find the optimal strategy to minimize its energy consumption and maximize the energy utilization of the UAV.

[0006] The technical solution of the present invention is as follows:

[0007] A method for collaborative search of unmanned aerial vehicles and unmanned ship captains based on reinforcement learning includes the following steps:

[0008] Step 1: Deploy multiple drones and unmanned ships in the mission area to form a collaborative search system and initialize it;

[0009] Step 2: Discretize the mission area into grids, and the drone and unmanned ship use a probabilistic graphical model to record the probability of target existence in each cell;

[0010] Step 3: The UAV is limited by its battery capacity during the mission. When the UAV is running low on energy, it sends a charging request to the UAV. The UAV plans a path based on the UAV battery energy and the probability of the target being there, minimizing the moving distance of the UAV.

[0011] Step 4. The unmanned ship uses a collaborative optimization scheduling algorithm based on a multi-agent deep deterministic policy gradient method to optimize its own motion trajectory, provide charging services for the drone and search for targets; if no surface target is found, re-execute step 2; if the drone does not have enough energy to complete the task, repeat steps 3-4 until the target position is identified through target probability.

[0012] Furthermore, in step 1, the task area is set to a length of , width is Rectangular area, deployment The drone was at an altitude of Flying over the mission area; An unmanned ship equipped with wireless charging equipment acts as a mobile charging station during the mission; Drones and The unmanned ships form a cross-domain unmanned cluster to collaboratively perform search tasks; the drone has an antenna for receiving energy, and the unmanned ship is equipped with a transmitting antenna; the drone monitors its own energy level, and when the drone has insufficient energy, it sends a charging request, the unmanned ship will approach the drone, the wireless charging device will start, and the unmanned ship will display that it is charging; information transmission between the drone and the unmanned ship, and between the unmanned ships, establishes an initial connection through a radio communication link.

[0013] Furthermore, in step 2, the task area is divided into A grid, Indicates the number of grids that the task area is divided into horizontally. Indicates the number of grids that the task area is divided into vertically; The center coordinates of the grid are , , , , Indicates The horizontal coordinates of the grid. Indicates The vertical coordinates of the grid; set Indicates There is a target in the grid or there is no target. Indicates There are targets in the grids. Indicates There is no target in the grid;

[0014] The UAV and the unmanned ship dynamically update the probability map of the mission area according to the Bayesian rule. The specific update formula is:

[0015] (1);

[0016] in, , Respectively represent time slots and time slot The generated probability map; and are the detection probability and false alarm probability respectively.

[0017] Furthermore, in step 3, the UAV is not able to complete the task due to the limitation of energy consumption and battery capacity during the probability map update process. Therefore, a remote wireless charging method is adopted. The unmanned ship transmits beamforming energy to the UAV. The energy obtained by the UAV depends on the distance between it and the unmanned ship, which is calculated as:

[0018] (2);

[0019] in, For distance calculation; and Respectively represent time slots Unmanned ship and drones The location coordinates of Indicates the flight altitude of the drone;

[0020] Drones in time slots Energy gained for:

[0021] (3);

[0022] in, is a binary variable; is the energy conversion efficiency; is the duration of a time slot; is the energy transmission power of the unmanned ship;

[0023] Assume that the maximum energy capacity of the drone is , then after the unmanned boat charging service, the drone will be in the time slot The remaining energy for:

[0024] (4);

[0025] in, For drones in time slots The remaining energy

[0026] The energy consumption of drones is divided into two categories: communication energy consumption and flight energy consumption. The communication energy consumption is fixed as , the flight energy consumption of the UAV during uniform flight for:

[0027] (5);

[0028] in, , , are constants related to different drones; and Respectively represent the speed and thrust of the drone;

[0029] Time slot Drone energy consumption for:

[0030] (6);

[0031] in, Express Derivation;

[0032] Drones in time slots The remaining energy is:

[0033] (7);

[0034] Drones In time slot The remaining energy satisfy ;in, The minimum reserved energy for the drone, then the drone Charging urgency Calculated as:

[0035] (8);

[0036] If the remaining energy of the current UAV is low, it has a high charging urgency, and the unmanned ship gives priority to serving the UAV with high charging urgency;

[0037] The movement of the unmanned ship is accompanied by its own energy consumption. The purpose of finding the optimal path is to minimize the average moving distance of the unmanned ship. Assuming that there is a linear relationship between energy consumption and moving distance, Defined as unmanned ship In time slot The moving distance at is expressed as:

[0038] (9);

[0039] in, Indicates time slot Unmanned ship The location coordinates of

[0040] Unmanned ship Energy consumption as follows:

[0041] (10);

[0042] in, is a coefficient; is the resistance of the unmanned ship when sailing on the sea; therefore unmanned ships until the total time slot Energy consumption for:

[0043] (11);

[0044] In the search, to optimize the motion trajectory of the unmanned ship, keep the energy of the drone above the threshold, and extend the working time of the drone, a series of parameters need to be optimized, including the motion energy consumption of the unmanned ship and the energy utilization rate of the drone. The optimization problem can be expressed as:

[0045] (12);

[0046] in, represents the expected value of the entire formula; represents the energy utilization rate of the UAV in the search mission, is a binary decision variable; is a constant; Indicates that the drone is in the time slot The area of ​​the search region at ; , , , For different constraints; constraints Indicates that the unmanned ship is in the time slot The distance moved is less than or equal to the maximum distance that can be moved ;constraint Indicates the range of the remaining energy of the drone; constraints Ensure that each drone is in its time slot Charged by at most one unmanned ship; Constraint Guaranteed unmanned ship and unmanned ships The distance between Greater than or equal to the safety distance to avoid collision.

[0047] Furthermore, in step 4, the unmanned ship cluster uses a multi-agent deep deterministic policy gradient algorithm to perform reinforcement learning based on the drone position, energy information, target probability information and state information of adjacent unmanned ships obtained in the current state, so as to complete action decision-making and optimize the trajectory; the specific process is:

[0048] In the unmanned ship cluster, each unmanned ship maintains its own Actor network and Critic network, defining , Unmanned Ship Actor network, Critic network, unmanned ship The Actor network parameters and Critic network parameters are and , the Actor-Critic method is used for centralized training and distributed execution. During training, the Critic network of each unmanned ship can obtain the strategy information of other unmanned ships;

[0049] By minimizing the loss function To update, the specific formula is as follows:

[0050] (13);

[0051] in, For The expected value of the sample in Represents the experience replay pool; It is a centralized action-value function; The status information of the unmanned ship at the current moment; For the current moment The set of actions taken by the unmanned ship, The unmanned ship at the current moment Actions taken; Indicates the target value at the current moment, , The unmanned ship at the current moment The reward value, is the discount factor, is the action value function of the next state, To perform the corresponding action back The next state information, For unmanned ships The action in the next state, For unmanned ships Observation information of express The set of target strategies used in the value function of the unmanned ship update; For unmanned ships The Actor network for the next state;

[0052] Perform gradient descent to update, the specific formula is:

[0053] (14);

[0054] in, Express Derivation; represents the optimization objective; To review the experience pool Randomly extract status information The expected value of Represents a state information randomly drawn from the experience replay pool; Experience replay pool By tuple express, ={ }for The rewards obtained by the unmanned ships are as follows: The unmanned ship at the current moment The reward value of The unmanned ship at the current moment Actions taken; For unmanned ships The current observation value; Express Derivation;

[0055] Soft update of unmanned ship Target Critic network parameters and target Actor network parameters , the specific formula is as follows:

[0056] (15);

[0057] in, Indicates a soft update operation; Represents the soft update coefficient.

[0058] Furthermore, after each unmanned ship performs an action, a total reward function is set as follows:

[0059] definition Unmanned ship In time slot The energy consumption reward when performing the corresponding task. The energy consumed by each unmanned ship is determined by the distance it moves. The formula is:

[0060] (16);

[0061] definition Unmanned ship In time slot The reward you get when charging the drone is as follows:

[0062] (17);

[0063] in, is a positive coefficient; when the unmanned ship receives charging requests from multiple drones, it will prioritize each drone according to its charging urgency; by evaluating the current drone energy level, the unmanned ship determines which drone has the highest charging urgency and gives priority to charging services for the drone with the highest charging urgency;

[0064] definition Unmanned ship In time slot The search reward guides the unmanned ship to explore high-probability areas within a limited time to obtain the highest profit; the search reward formula is:

[0065] (18);

[0066] If the drone ship cannot charge any drone in time and the remaining battery energy is less than , then use Indicates the penalty for drones; definition Unmanned ship and unmanned ships The penalty for a collision between them is as follows:

[0067] (19);

[0068] (20);

[0069] In summary, the total reward function is expressed as follows:

[0070] (twenty one);

[0071] in, For unmanned ships In time slot Total reward.

[0072] Furthermore, the specific process of the collaborative optimization scheduling algorithm based on the multi-agent deep deterministic policy gradient method is as follows:

[0073] Step 4.1, initialization parameters, including , , , , , , ; is the number of rounds;

[0074] Step 4.2: Perform an external round loop; the specific process is as follows:

[0075] Step 4.2.1, randomly assign the positions of the drone and the unmanned ship;

[0076] Step 4.2.2: Initialize the time slot A random process , for motion exploration;

[0077] Step 4.2.3, taking the unmanned ship as an intelligent agent, obtain the initial observation values ​​of all intelligent agents;

[0078] Step 4.2.4: Perform an internal time step cycle; the specific process is as follows:

[0079] Step 4.2.4.1. Each unmanned ship selects an action according to the current strategy. Suppose the unmanned ship at the current moment The actions taken are: ;

[0080] Step 4.2.4.2: Execute action set , get the next state information and A collection of rewards for unmanned ships ;

[0081] Step 4.2.4.3. Store to experience replay pool middle;

[0082] Step 4.2.4.4, update observations: ;

[0083] Step 4.2.4.5, perform internal loop;

[0084] Step 4.2.4.5.1. From the Experience Replay Pool Randomly draw small batches of samples from ;in, The unmanned ship at the current moment Status information; The unmanned ship at the current moment Actions taken; The unmanned ship at the current moment The reward value of For unmanned ships The next state information after executing the action;

[0085] Step 4.2.4.5.2, update the critic network parameters of each unmanned ship according to the minimized loss function of formula (13);

[0086] Step 4.2.4.5.3, perform gradient descent according to formula (14) to update the Actor network parameters of each unmanned ship;

[0087] Step 4.2.4.6: If each agent has completed the update, the inner loop ends;

[0088] Step 4.2.4.7, update the target Critic network parameters and target Actor network parameters of each unmanned ship according to formula (15);

[0089] Step 4.2.5: If the optimization scheduling time is exceeded, end the internal time step loop;

[0090] Step 4.3: If the number of steps in the outer loop is reached, end the outer round loop.

[0091] The beneficial technical effects brought by the present invention: The present invention aims at the characteristics of collaborative search scenarios in marine environments, utilizes the complementary advantages of UAVs and USVs, forms a cross-domain unmanned cluster to collaboratively perform search tasks, and obtains target information in the mission area through a probabilistic graph model. Taking into account the energy limitation of UAVs during the mission, USVs are used as mobile charging stations to plan paths according to different influencing factors (such as UAV energy level, target existence probability, etc.). A trajectory optimization problem is designed to improve the probability of target detection and maximize the energy utilization of UAVs and minimize the energy consumption of USVs. At the same time, several key factors such as UAV position, UAV energy information, target probability information, and status information of adjacent unmanned ships are considered to optimize the trajectory of the unmanned ship. On this basis, an unmanned ship optimization scheduling algorithm based on multi-agent reinforcement learning is proposed to find the optimal solution to the optimization problem to achieve heterogeneous long-endurance collaborative search of UAVs and unmanned ships in large-scale marine environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 This is a flow chart of the reinforcement learning-based collaborative search method for drones and unmanned ship captains of the present invention.

[0093] Figure 2 This is a schematic diagram of the invention of charging a UAV on demand in a mission area through an unmanned boat mobile charging station.

[0094] Figure 3 The figure is a schematic diagram of the energy replenishment process of a UAV of the present invention.

[0095] Figure 4 It is a schematic diagram of the framework of the reinforcement learning-based collaborative search method for the endurance of a UAV and an unmanned ship captain of the present invention.

[0096] Figure 5 It is a schematic diagram of updating the target probability map of the task area in the simulation experiment of the present invention.

[0097] Figure 6 Schematic diagram of the energy level change of the UAV during the mission execution in the simulation experiment of the present invention.

[0098] Figure 7 This is a comparison chart of energy consumption of unmanned boats serving different numbers of drones in the simulation experiment of the present invention.

[0099] Figure 8 This is a comparison chart of the average charging energy of multi-agent deep reinforcement learning, online auction method and fixed base station charging method under different numbers of drones in the simulation experiment of the present invention.

[0100] Fig. 9 This is a comparison chart of drone energy utilization using multi-agent deep reinforcement learning, online auction method, and fixed base station charging method in the simulation experiment of the present invention. DETAILED DESCRIPTION

[0101] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0102] In the present invention, USV acts as a mobile charging station to provide wireless charging service for UAVs, and increases the duration of collaborative search tasks under uninterrupted search execution. Firstly, the USV trajectory optimization problem under target search constraints is proposed to minimize the energy consumption of the unmanned ship and maximize the energy utilization of the UAV. Then, the problem is modeled as a partially observable Markov decision process (POMDP), and an optimization scheduling algorithm using the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) method is designed for heterogeneous collaborative long-endurance search tasks between UAVs and unmanned ships under energy constraints.

[0103] The present invention adopts a method based on multi-agent deep reinforcement learning. The unmanned ship plans the optimal path according to the battery energy of the drone and the probability of target existence, minimizing its moving distance. At the same time, it considers the location information of the drone and the status information of adjacent unmanned ships to optimize its motion trajectory, provide charging services for the drone and perform target search.

[0104] like Figure 1 As shown, the present invention specifically includes the following steps:

[0105] Step 1: Deploy multiple drones and unmanned ships in the mission area to form a collaborative search system and initialize it;

[0106] like Figure 2 As shown, set the task area to a length of , width is Rectangular area, deployment The drone was at an altitude of Flying over the mission area; An unmanned ship equipped with wireless charging equipment acts as a mobile charging station during the mission; Drones and The unmanned ships form a cross-domain unmanned cluster to perform search tasks in coordination. Assume that there is an antenna for receiving energy on the drone and the unmanned ship is equipped with a transmitting antenna. The drone monitors its own energy level. When the drone is low on energy, it sends a charging request. The unmanned ship will approach the drone, the wireless charging device will start, and the unmanned ship will display that it is charging. The information transmission between the drone and the unmanned ship and between the unmanned ships is initially connected through a radio communication link.

[0107] Step 2: Discretize the mission area into grids. The drone and unmanned ship use a probabilistic graphical model to record the probability of the target existence in each cell. The specific process is as follows:

[0108] Divide the task area into A grid, Indicates the number of grids that the task area is divided into horizontally. Indicates the number of grids that the task area is divided into vertically. The center coordinates of the grid are , , , , Indicates The horizontal coordinates of the grid. Indicates The vertical coordinates of the grid. Indicates There is a target in the grid or there is no target. Indicates There are targets in the grids. Indicates There is no target in the grid. The UAV and the unmanned ship dynamically update the probability map of the mission area according to the Bayesian rule. The specific update formula is:

[0109] (1);

[0110] in, , Respectively represent time slots and time slot The generated probability map; and are the detection probability and false alarm probability, respectively. The UAV tends to update the probability map quickly, while the unmanned ship prefers to search the area with high probability first.

[0111] Step 3: The UAV is limited by its battery capacity during the mission. When the UAV is running low on energy, it sends a charging request to the unmanned ship. The unmanned ship plans a path based on the UAV battery energy and the target existence probability to minimize its moving distance. The specific implementation is as follows:

[0112] The energy consumption and battery capacity of the drone are limited in the process of updating the probability map, which is not enough to support the task completion. Therefore, the remote wireless charging method is adopted. The energy transmission between the unmanned ship and the drone is as follows: Figure 3 The unmanned ship transmits beamforming energy to the drone. The energy obtained by the drone depends on the distance between it and the unmanned ship, which can be calculated as:

[0113] (2);

[0114] in, For distance calculation; and Respectively represent time slots Unmanned ship and drones The location coordinates of represents the flight altitude of the drone. Energy gained for:

[0115] (3);

[0116] in, is a binary variable, if , then no one boat In time slot UAV Charging is performed, otherwise it is 0; is the energy conversion efficiency; is the duration of a time slot; is the energy transmission power of the unmanned ship. Assume that the maximum energy capacity of the UAV is , then after the unmanned boat charging service, the drone will be in the time slot The remaining energy for:

[0117] (4);

[0118] in, For drones in time slots The remaining energy

[0119] The energy consumption of drones is mainly divided into two categories: communication energy consumption and flight energy consumption. Since the communication energy consumption is much lower than the flight energy consumption, for the convenience of calculation, the communication energy consumption is fixed as The flight energy consumption of the drone during uniform flight for:

[0120] (5);

[0121] in, , , are constants related to different drones; and represent the speed and thrust of the drone respectively. Therefore, at the time slot Drone energy consumption for:

[0122] (6);

[0123] in, Express Derivation;

[0124] Therefore, the remaining energy of the drone in each time slot is:

[0125] (7);

[0126] in, For drones in time slots The remaining energy

[0127] Drones In time slot The remaining energy satisfy .in, The minimum reserved energy for the drone, then the drone Charging urgency It can be calculated as:

[0128] (8);

[0129] If the remaining energy of the current UAV is low, it has a higher charging urgency, so the unmanned ship gives priority to serving the UAV with high charging urgency.

[0130] The movement of the unmanned ship is accompanied by its own energy consumption. The purpose of finding the optimal path is to minimize the average moving distance of the unmanned ship. Assuming that there is a linear relationship between energy consumption and moving distance, Defined as unmanned ship In time slot The moving distance at can be expressed as:

[0131] (9);

[0132] in, Indicates time slot Unmanned ship The location coordinates of

[0133] Therefore, unmanned ships Energy consumption It can be written as follows:

[0134] (10);

[0135] in, is a coefficient; is the resistance of the unmanned ship when sailing on the sea, so unmanned ships until the total time slot Energy consumption for:

[0136] (11);

[0137] In the search, the USV's motion trajectory is optimized to keep the drone's energy above the threshold and extend the drone's working time. Therefore, a series of parameters need to be optimized, including the unmanned ship's motion energy consumption and the drone's energy utilization rate. The optimization problem is expressed as:

[0138] (12);

[0139] in, represents the expected value of the entire formula; represents the energy utilization rate of the UAV in the search mission, is a binary decision variable, which is set to 1 when the residual energy is greater than the threshold, otherwise it is set to 0; is a constant; Indicates that the drone is in the time slot The area of ​​the search region at ; , , , For different constraints. Constraints Indicates that the unmanned ship is in the time slot The distance moved is less than the maximum distance that can be moved .constraint Indicates the range of the remaining energy of the drone. Ensure that each drone is in its time slot Can be charged by at most one unmanned ship. Constraints Ensure unmanned ship and unmanned ships The distance between Greater than or equal to the safety distance to avoid collision.

[0140] Step 4: The unmanned ship optimizes its own motion trajectory using a collaborative optimization scheduling algorithm based on a multi-agent deep deterministic policy gradient method, provides charging services for the drone and searches for targets; if no surface target is found, step 2 is re-executed; if the drone has insufficient energy to complete the task, steps 3-4 are repeated until the target position is identified by target probability. The framework diagram of the collaborative search method for drones and unmanned ships based on reinforcement learning is shown in the figure below: Figure 4 As shown, the specific implementation is:

[0141] In order to solve the problem that drones need charging services due to energy constraints in collaborative search tasks, a collaborative optimization scheduling algorithm is designed. The collaborative optimization scheduling algorithm is specifically based on the multi-agent deep deterministic policy gradient algorithm (MADDPG). The unmanned ship cluster uses the multi-agent deep deterministic policy gradient algorithm (MADDPG) to perform reinforcement learning based on the drone position, energy information, target probability information, and state information of adjacent unmanned ships obtained from the current state, so as to complete action decisions and ultimately optimize the trajectory. In the unmanned ship cluster, each unmanned ship maintains its own Actor network and Critic network, and defines , Unmanned Ship Actor network, Critic network, unmanned ship The Actor network parameters and Critic network parameters are and , the Actor-Critic method is used for centralized training and distributed execution. During training, the Critic network of each unmanned ship can obtain the strategy information of other unmanned ships.

[0142] Unmanned ship Critic network parameters By minimizing the loss function To update, the specific formula is as follows:

[0143] (13);

[0144] in, For The expected value of the sample in represents the average of all possible states, actions, and rewards; Represents the experience replay pool, which is used to store data; It is a centralized action-value function; The status information of the unmanned ship at the current moment; For the current moment The set of actions taken by the unmanned ship, The unmanned ship at the current moment Actions taken; represents the target value at the current moment, , The unmanned ship at the current moment The reward value, is the discount factor, is the action value function of the next state, To perform the corresponding action back The next state information, For unmanned ships The action in the next state, For unmanned ships Observation information of express The set of target strategies used in the value function of the unmanned ship update; For unmanned ships Actor network for the next state.

[0145] Unmanned ship Actor network parameters Perform gradient descent to update, the specific formula is:

[0146] (14);

[0147] in, Express Derivation; represents the optimization objective; To review the experience pool Randomly extract status information The expected value of Represents a state information randomly drawn from the experience replay pool; Experience replay pool By tuple express, ={ }for The rewards obtained by the unmanned ships are as follows: The unmanned ship at the current moment The reward value of Indicates that the Critic network is in the unmanned ship According to the current strategy Actions taken The value function value of . For unmanned ships The current observation value; Express Derivation; The unmanned ship at the current moment Actions taken;

[0148] Soft update of unmanned ship Target Critic network parameters and target Actor network parameters , the specific formula is as follows:

[0149] (15);

[0150] in, Indicates a soft update operation; Represents the soft update coefficient.

[0151] After each unmanned ship performs an action, a total reward function is set as follows:

[0152] definition Unmanned ship In time slot The energy consumption reward when performing the corresponding task, the energy consumed by each unmanned ship is determined by the distance it moves. Therefore, the energy consumption reward is expressed as:

[0153] (16);

[0154] definition Unmanned ship In time slot The reward you get when charging the drone is as follows:

[0155] (17);

[0156] in, is a positive coefficient representing a basic reward designed to encourage drones to charge regardless of the charging outcome. When the unmanned ship receives charging requests from multiple drones, it prioritizes each drone based on its charging urgency. By evaluating the current drone energy levels, the unmanned ship determines which drone has the highest charging urgency and prioritizes charging services accordingly.

[0157] definition Unmanned ship In time slot The search reward guides the unmanned ship to explore high-probability areas within a limited time to obtain the highest profit. The search reward formula is:

[0158] (18);

[0159] If the drone ship cannot charge any drone in time and the remaining battery energy is less than , then use Indicates a penalty for a drone. Definition Unmanned ship and unmanned ships The penalty for collision between them. The specific formula is as follows:

[0160] (19);

[0161] (20);

[0162] In summary, the total reward function can be expressed as follows:

[0163] (twenty one);

[0164] in, For unmanned ships In time slot Total reward.

[0165] The specific implementation steps of the collaborative optimization scheduling algorithm based on the multi-agent deep deterministic policy gradient method described in step 4 are as follows:

[0166] Step 4.1, initialization parameters, including , , , , , , ; is the number of rounds;

[0167] Step 4.2: Perform an external round loop; the specific process is as follows:

[0168] Step 4.2.1, randomly assign the positions of the drone and the unmanned ship;

[0169] Step 4.2.2: Initialize the time slot A random process , for motion exploration;

[0170] Step 4.2.3, taking the unmanned ship as an intelligent agent, obtain the initial observation values ​​of all intelligent agents;

[0171] Step 4.2.4: Perform an internal time step cycle; the specific process is as follows:

[0172] Step 4.2.4.1. Each unmanned ship selects an action according to the current strategy. Suppose the unmanned ship at the current moment The actions taken are: ;

[0173] Step 4.2.4.2: Execute action set , get the next state information and A collection of rewards for unmanned ships ;

[0174] Step 4.2.4.3. Store to experience replay pool middle;

[0175] Step 4.2.4.4, update observations: ;

[0176] Step 4.2.4.5, perform internal loop;

[0177] Step 4.2.4.5.1. From the Experience Replay Pool Randomly draw small batches of samples from ;in, The unmanned ship at the current moment Status information; The unmanned ship at the current moment Actions taken; The unmanned ship at the current moment The reward value of For unmanned ships The next state information after executing the action;

[0178] Step 4.2.4.5.2, update the critic network parameters of each unmanned ship according to the minimized loss function of formula (13);

[0179] Step 4.2.4.5.3, perform gradient descent according to formula (14) to update the Actor network parameters of each unmanned ship;

[0180] Step 4.2.4.6: If each agent has completed the update, the inner loop ends;

[0181] Step 4.2.4.7, update the target Critic network parameters and target Actor network parameters of each unmanned ship according to formula (15);

[0182] Step 4.2.5: If the optimization scheduling time is exceeded, end the internal time step loop;

[0183] Step 4.3: If the number of steps in the outer loop is reached, end the outer round loop.

[0184] The effect of the present invention is further illustrated by the following simulation experiment:

[0185] (1) The simulation conditions are as follows:

[0186] The simulation experiment was conducted using Python 3.9 and MATLAB 2022a. Four fully connected neural networks were deployed using Pytorch, each network containing two hidden layers, with a total of 64 units. In the simulation, drones and unmanned ships were deployed over a target area of ​​10km×10km. The UAVs are flying or hovering at a fixed height of 10 meters, and the remaining energy of the UAVs is uniformly distributed between [30%, 100%]. In the initial stage, the positions of the UAVs and the UAVs are randomly assigned, and the values ​​of the relevant parameters are set as follows:

[0187] The search range of the UAV is set to 300m; the maximum communication distance between two adjacent unmanned ships is set to 100m; the maximum energy capacity of the UAV is set to 97.85Wh; the minimum reserved energy of the UAV is set to 20%; the energy transmission power of the unmanned ship is set to 80W; the energy conversion efficiency is set to 0.9; the discount factor is set to 0.95; the learning rate is set to 0.01; the update rate of the target network is set to 0.01; the length of the experience replay pool is set to ; The batch size is set to 1024; The number of rounds is set to 20000.

[0188] (2) The simulation results are as follows:

[0189] First, verify the effectiveness of drones and unmanned ships in searching for targets in the mission area. Figure 5 The target existence probability map generated by the UAV and the USV in the search mission area within 100 time steps is shown. In the initial stage, the prior probability is randomly distributed, and the color depth of each grid represents the probability of the target existence. In the areas explored by the UAV and USV, the darkening of the grid color indicates that the probability of the target existence in these areas is reduced. Grids with higher target existence probabilities will attract the USV cluster, thereby increasing the possibility of searching for the target. At the same time, the UAV cluster will continue to update the probability map to optimize the search strategy.

[0190] In order to observe the changes in the residual energy of the UAV during the mission execution, the experimental results of 5 UAVs within 100 time steps were used, such as Figure 6 As shown in the figure. The initial energy level follows a uniform distribution of [30%, 100%], and the energy demand of the UAV fluctuates as the mission progresses. Once the energy of some UAVs drops to an insufficient state, they will issue a charging request. The USVs coordinate scheduling to charge these UAVs immediately, but the energy of some UAVs will not increase to 100%. This is because in the simulation environment, each UAV works at the same time and the number of USVs is less than the number of UAVs. The scheduling algorithm between USVs first provides charging services for low-energy UAVs. In addition, it can be found that in some time slots, the total energy of the UAV drops rapidly. However, the proposed scheduling algorithm effectively prevents this downward trend and maintains the total energy above the minimum threshold of 20%.

[0191] In order to show the energy consumption of USV during mobile dispatching, Figure 7The figure shows the motion energy consumption of USV under different numbers of UAVs. The discrete points in the figure represent the energy consumption at different time steps, and the diamond discrete points represent the energy consumption at the first time step. From the data in the figure, it can be seen that the energy consumption of USV shows a trend of first rising and then falling. Since the positions of UAVs and USVs are randomly distributed in the initial stage, and the charging service requirements of each UAV are different, USVs need to continuously learn scheduling strategies. After that, USVs gradually learn the optimal moving direction through collaborative communication to minimize energy consumption while meeting the energy requirements of UAVs. In addition, it can be seen from the figure that with the increase in the number of UAVs, the energy consumption of USVs is also increasing. This is because the number of objects that need to be served increases, and the moving distance of USVs increases, resulting in increased energy consumption.

[0192] The average charging energy of multi-agent deep reinforcement learning (i.e., the collaborative optimization scheduling algorithm based on the multi-agent deep deterministic policy gradient method proposed in this invention, which can also be referred to as the scheduling algorithm based on multi-agent reinforcement learning), online auction method and fixed base station charging method under different numbers of drones was further verified. Figure 8 As shown in the figure, compared with the online auction method, the proposed scheduling algorithm based on multi-agent reinforcement learning shows a more superior effect when the number of drones increases, and the average charging energy is greater than the online auction method. However, although the charging method based on fixed charging stations has increased the average charging energy, the risk of mission interruption increases as the energy of the drones continues to decrease during the return process. The most significant advantage of using unmanned ships to charge drones is that drones can be charged while performing tasks, thus saving the time required to return to the charging station.

[0193] Fig. 9 The energy utilization of UAVs under multi-agent deep reinforcement learning, online auction method and fixed base station charging method was verified. The results show that the energy utilization distribution of the scheduling algorithm based on multi-agent reinforcement learning is more concentrated, while the data distribution of the other two methods is relatively scattered. The scheduling algorithm based on multi-agent reinforcement learning and the mobile charging method of the online auction algorithm keep the battery level of the UAV to run the task continuously, while the charging method of the fixed charging station may cause the energy of the UAV to be lower than the threshold and need to return to the shore base station for charging, resulting in zero energy utilization in the process and large fluctuations. Therefore, compared with the other two methods, the scheduling algorithm based on multi-agent reinforcement learning has a better energy utilization, which ensures the continuous operation of the UAV in the search task and a more stable energy utilization distribution.

[0194] The above numerical results show that the scheme of the present invention has good performance, can effectively provide wireless charging services for drones in search missions, extend the collaborative search time, and maximize the energy utilization of drones and reduce the motion energy consumption of unmanned ships.

[0195] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for collaborative search of drone and unmanned ship endurance based on reinforcement learning, characterized in that: The steps include: Step 1: Deploy multiple drones and unmanned ships in the mission area to form a collaborative search system and initialize it; Step 2: Discretize the mission area into grids, and the drone and unmanned ship use a probabilistic graphical model to record the probability of target existence in each cell; Step 3: The UAV is limited by its battery capacity during the mission. When the UAV is running low on energy, it sends a charging request to the UAV. The UAV plans a path based on the UAV battery energy and the probability of the target being there, minimizing the moving distance of the UAV. In step 3, the UAV is not able to complete the task due to the limitation of energy consumption and battery capacity during the probability map update process. Therefore, a remote wireless charging method is adopted. The unmanned ship transmits beamforming energy to the UAV. The energy obtained by the UAV depends on the distance between it and the unmanned ship, which is calculated as: in, For distance calculation; and They represent the position coordinates of unmanned ship i and unmanned aerial vehicle m at time slot t respectively; h represents the flight altitude of the unmanned aerial vehicle; The energy obtained by the drone in time slot t for: Among them, ψ i,m (t) is a binary variable; 0<ξ<1 is the energy conversion efficiency; δ is the duration of a time slot; P c is the energy transmission power of the unmanned ship; Assume that the maximum energy capacity of the drone is E max , then after the unmanned boat charging service, the remaining energy of the drone in time slot t for: in, is the remaining energy of the UAV at time slot t-1; The energy consumption of drones is divided into two categories: communication energy consumption and flight energy consumption. The communication energy consumption is fixed as P com , the flight energy consumption of the drone during uniform flight P f for: Among them, κ1, κ2, κ3 are different UAV related constants; v and Respectively represent the speed and thrust of the drone; Energy consumption E of drone in time slot t m (t) is: Among them, dt represents the derivative with respect to t; The remaining energy of the drone at time slot t+1 is: The remaining energy of drone m at time slot t satisfy Among them, E th The minimum reserved energy of the drone is, then the charging urgency of drone m Calculated as: If the remaining energy of the current UAV is low, it has a high charging urgency, and the unmanned ship gives priority to serving the UAV with high charging urgency; The movement of the unmanned ship is accompanied by its own energy consumption. The purpose of finding the optimal path is to minimize the average moving distance of the unmanned ship. Assuming that there is a linear relationship between energy consumption and moving distance, It is defined as the moving distance of unmanned ship i at time slot t, expressed as: in, represents the position coordinates of unmanned ship i at time slot t-1; Energy consumption of unmanned ship i as follows: Among them, ρ is a coefficient; R es is the resistance of the unmanned ship when sailing on the sea; therefore, the energy consumption E of N unmanned ships until the total time slot T S for: In the search, to optimize the motion trajectory of the unmanned ship, keep the energy of the drone above the threshold, and extend the working time of the drone, a series of parameters need to be optimized, including the motion energy consumption of the unmanned ship and the energy utilization rate of the drone. The optimization problem can be expressed as: in, represents the expected value of the entire formula; represents the energy utilization rate of the UAV in the search task, β is a binary decision variable; c is a constant; S t represents the search area of ​​the drone at time slot t; C1, C2, C3, and C4 are different constraints; constraint C1 means that the distance the drone moves at time slot t is less than or equal to the maximum distance l that it can move max ; Constraint C2 represents the range of the remaining energy of the drone; Constraint C3 ensures that each drone is charged by at most one drone ship in time slot t; Constraint C4 ensures that the distance between drone ship i and drone ship j Greater than or equal to the safety distance To avoid collision; Step 4: The unmanned ship optimizes its own motion trajectory using a collaborative optimization scheduling algorithm based on a multi-agent deep deterministic policy gradient method, provides charging services for the drone, and searches for targets; if no surface target is found, step 2 is executed again; if the drone does not have enough energy to complete the task, steps 3-4 are repeated until the target position is identified through target probability; In step 4, the unmanned ship cluster uses a multi-agent deep deterministic policy gradient algorithm to perform reinforcement learning based on the drone position, energy information, target probability information and state information of adjacent unmanned ships obtained in the current state, so as to complete action decision-making and optimize the trajectory; the specific process is: In the unmanned ship cluster, each unmanned ship maintains its own Actor network and Critic network, defining They are the Actor network and Critic network of unmanned ship i, and the Actor network parameters and Critic network parameters of unmanned ship i are θ i and ω i , the Actor-Critic method is used for centralized training and distributed execution. During training, the Critic network of each unmanned ship can obtain the strategy information of other unmanned ships; ω i By minimizing the loss function Loss(ω i ) to update, the specific formula is as follows: in, is the expected value of the samples in D; D represents the experience replay pool; is a centralized action-value function; x is the state information of the unmanned ship at the current moment; a=(a1,a2,...,a N ) is the set of actions taken by N unmanned ships at the current moment, a N The action taken by the unmanned ship N at the current moment; Indicates the target value at the current moment, r i is the reward value of unmanned ship i at the current moment, γ is the discount coefficient, is the action value function of the next state, x′ is the corresponding action a′=(a′1,a′2,...,a j ′,...,a′ N ) The next state information of x after , a′ j is the action of unmanned ship j in the next state, o j is the observation information of unmanned ship j; Represents the set of target strategies used in the value update function of N unmanned ships; is the Actor network of the next state of unmanned ship j; θ i Perform gradient descent to update, the specific formula is: in, Represents the i Derivative; J(·) represents the optimization objective; is the expected value of a randomly selected state information x from the experience replay pool D, x~D represents a state information randomly selected from the experience replay pool; the experience replay pool D consists of a tuple (x, x′, a1,…, a N ,r1,…,r N ) means, r={r1,…,r N } is the reward set obtained by N unmanned ships, r N is the reward value of the unmanned ship N at the current moment; a i is the action taken by unmanned ship i at the current moment; o i is the current observation value of unmanned ship i; Indicates a i Derivation; Soft update of target critic network parameters ω of unmanned ship i i ′ and target Actor network parameters θ i ′, the specific formula is as follows: Among them, ← represents the soft update operation; ε represents the soft update coefficient; After each unmanned ship performs an action, a total reward function is set as follows: definition It represents the energy consumption reward when the unmanned ship i performs the corresponding task in time slot t. The energy consumed by each unmanned ship is determined by the distance it moves. The formula is: definition It represents the reward obtained by unmanned ship i when charging the drone at time slot t. The formula is as follows: in, is a positive coefficient; when the unmanned ship receives charging requests from multiple drones, it will prioritize each drone according to its charging urgency; by evaluating the current drone energy level, the unmanned ship determines which drone has the highest charging urgency and gives priority to charging services for the drone with the highest charging urgency; definition represents the search reward of unmanned ship i in time slot t. The search reward guides the unmanned ship to explore the high-probability area within a limited time to obtain the highest profit. The search reward formula is: If the unmanned ship cannot charge any drone in time and the remaining battery energy is lower than E th , then use R l Represents the penalty for the drone; define R d It represents the penalty when a collision occurs between unmanned ship i and unmanned ship j. The specific formula is as follows: In summary, the total reward function is expressed as follows: in, is the total reward of unmanned ship i in time slot t; The specific process of the collaborative optimization scheduling algorithm based on the multi-agent deep deterministic policy gradient method is as follows: Step 4.1: Initialize parameters, including θ i ,ω i ,ω i ′、θ i ′, D, T; is the number of rounds; Step 4.2: Perform an external round loop; the specific process is as follows: Step 4.2.1, randomly assign the positions of the drone and the unmanned ship; Step 4.2.2: Initialize a random process in time slot t For motion exploration; Step 4.2.3, taking the unmanned ship as an intelligent agent, obtain the initial observation values ​​of all intelligent agents; Step 4.2.4: Perform an internal time step cycle; the specific process is as follows: Step 4.2.4.

1. Each unmanned ship selects an action according to the current strategy. Suppose the action taken by unmanned ship i at the current moment is: Step 4.2.4.2, execute action set a = (a1, a2, ..., a N ), obtain the next state information x′ and the reward set r=(r1,r2,...,r N ); Step 4.2.4.

3. Store (x, a, r, x′) into the experience replay pool D. Step 4.2.4.4, update the observation value: x←x′; Step 4.2.4.5, perform internal loop; Step 4.2.4.5.

1. Randomly extract a small batch of samples (x j ,a j ,r j ,x′ j ), where x j is the state information of unmanned ship j at the current moment; a j is the action taken by unmanned ship j at the current moment; r j is the reward value of unmanned ship j at the current moment; x′ j is the next state information after the unmanned ship j performs the action; Step 4.2.4.5.2, update the critic network parameters of each unmanned ship according to the minimized loss function of formula (13); Step 4.2.4.5.3, perform gradient descent according to formula (14) to update the Actor network parameters of each unmanned ship; Step 4.2.4.6: If each agent has completed the update, the inner loop ends; Step 4.2.4.7, update the target Critic network parameters and target Actor network parameters of each unmanned ship according to formula (15); Step 4.2.5: If the optimization scheduling time is exceeded, end the internal time step loop; Step 4.3: If the number of steps in the outer loop is reached, end the outer round loop.

2. The method for collaborative search of unmanned aerial vehicles and unmanned ship captain endurance based on reinforcement learning according to claim 1 is characterized in that: In step 1, the task area is set to a length of Width In a rectangular area, M drones are deployed to fly over the mission area at a height of h; N unmanned ships are equipped with wireless charging equipment to act as mobile charging stations during the mission; M drones and N unmanned ships form a cross-domain unmanned cluster to collaboratively perform the search mission; The drone has an antenna for receiving energy, and the unmanned ship is equipped with a transmitting antenna; the drone monitors its own energy level, and when the drone's energy is insufficient, it sends a charging request, and the unmanned ship will approach the drone, the wireless charging device will start, and the unmanned ship will display that it is charging; information transmission between the drone and the unmanned ship and between unmanned ships establishes an initial connection through a radio communication link.

3. The method for collaborative search of unmanned aerial vehicles and unmanned ship captain endurance based on reinforcement learning according to claim 1 is characterized in that: In step 2, the task area is divided into L x ×W y grid, L x Indicates the number of grids that the task area is divided into horizontally, W y Indicates the number of grids divided into the task area in the vertical direction; the center coordinate of the kth grid is g k =(x k ,y k ), x k ∈{1,2,...,L x },y k ∈{1,2,...,W y }, k∈{1,2,...,L x ×W y }, x k Represents the horizontal coordinate of the kth grid, y k Represents the vertical coordinate of the kth grid; set θ k Indicates whether there is a target in the kth grid or not, θ k =1 means there is a target in the kth grid, θ k =0 means there is no target in the kth grid; The UAV and the unmanned ship dynamically update the probability map of the mission area according to the Bayesian rule. The specific update formula is: Among them, Γ k (t),Γ k (t-1) represents the probability graph generated at time slot t and time slot t-1 respectively; and f are the detection probability and false alarm probability respectively.

Images