Underwater multi-AUV task allocation method and device based on deep reinforcement learning

Abstract

The invention discloses an underwater multi-AUV task allocation method based on deep reinforcement learning. The underwater multi-AUV task allocation method comprises the following steps: carrying out gridding processing on an underwater region; establishing an optimal distribution strategy based on a deep reinforcement learning algorithm, and determining sensor nodes to be collected by each AUV and access paths along the nodes; setting a target function to be minimum in total task completion path time delay, and setting a current optimal task allocation strategy as a strategy phi; starting from the first node, nodes are distributed randomly according to the probability of epsilon, and nodes are distributed according to the strategy phi according to the probability of 1-epsilon; repeating the process until the nodes are distributed completely; calculating the driving time delay of the AUV between the two nodes, and defining the driving time delay as side time delay; defining an AUV access node sequence as an asymmetric TSP problem and solving the problem by adopting an LKH algorithm; calculating an edge delay matrix of the AUV sailing among all the nodes and inputting the edge delay matrix into an optimal distribution strategy; and generating an initial strategy phi by using a GA (Genetic Algorithm), and iteratively optimizing the strategy phi until an optimal solution of the target function is obtained. According to the invention, the collection time delay of multiple AUVs is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater wireless sensor networks, and in particular to an underwater multi-AUV task allocation method and device based on deep reinforcement learning. Background Art

[0002] At present, underwater wireless sensor networks (UWSN) are composed of nodes with underwater acoustic communication capabilities and computing capabilities. Due to its real-time, convenient and easy-to-expand advantages, it has been widely used in underwater early warning, target detection, environmental monitoring and other fields in recent years, helping people to carry out systematic monitoring and research on the ocean. It is generally composed of various sensor nodes and other devices. Underwater sensor nodes perceive the surrounding environment and collect corresponding data. After processing, these data are forwarded to neighboring sensor nodes or surface aggregated sensor nodes in a timely manner to provide rapid and necessary information for upper-level application decisions. However, due to the uncertainty and complexity of the underwater environment, low-latency data collection in UWSN is a very challenging task.

[0003] The application scenario of the present invention is an underwater wireless sensor network that assists in collecting data with the assistance of multiple autonomous underwater vehicles (AUVs). Considering that underwater sensor nodes and AUVs are mostly arranged in a real ocean environment, the complexity of ocean currents and the difficulty of multiple AUVs in reasonably cooperating to collect data bring natural challenges to low-latency data collection. This challenge has two specific aspects: on the one hand, the complex ocean currents in the ocean environment will hinder the navigation of AUVs between sensor nodes and increase the navigation delay, and even when the ocean current speed is too large, the AUV will not be able to move forward; on the other hand, the underwater sensor nodes are in different underwater positions and the amount of data generated. Unreasonable AUV task allocation will not be able to exert the collaborative ability of multiple AUVs, nor can it reduce the collection delay. Therefore, how to avoid AUVs from being affected by ocean currents or even using ocean currents during navigation, how to reasonably assign tasks to multiple AUVs, and ultimately reduce the data collection delay is a very interesting challenge.

[0004] In recent years, data collection in underwater sensor networks has been widely discussed. There are mainly two types: multi-hop transmission and AUV-assisted collection. Multi-hop transmission means that the data of sensor nodes deep in the seabed is transmitted to the upper nodes hop by hop. However, this means that the nodes closer to the water surface receive and forward more data, and consume more energy. That is, the energy consumption of nodes is uneven, and some nodes are prone to "die" prematurely. Using AUV clusters to assign tasks and then cooperate to collect data from underwater sensor nodes is an effective way to solve this problem. However, there is a lack of research on multi-AUV task allocation algorithms in ocean current environments in the field of underwater sensor networks. Other existing studies only assign tasks based on the Euclidean distance between AUVs and sensor nodes, without considering the impact of ocean currents on data acquisition delays. At the same time, the task allocation algorithms in previous studies themselves also have shortcomings such as insufficient optimization capabilities and easy to fall into local optimality. Summary of the invention

[0005] In view of the above problems, the present invention proposes an underwater multi-AUV task allocation method and device based on deep reinforcement learning, which can significantly improve the delay of data acquisition.

[0006] The technical solution adopted by the present invention to solve the technical problems existing in the known technology is:

[0007] An underwater multi-AUV task allocation method based on deep reinforcement learning is proposed. The underwater area is gridded and filled with real ocean current data. The AUV is assumed to sail between the grids. An optimization allocation strategy based on the deep reinforcement learning algorithm is established to allocate tasks to multiple AUVs, determine the sensor nodes that each AUV needs to collect data and the access path along the sensor nodes. The objective function is set to minimize the path delay for the completion of the total task, and the current optimal task allocation strategy is set as strategy φ. Starting from the first sensor node, the sensor nodes are randomly allocated with a probability of ε, and the sensor nodes are allocated according to strategy φ with a probability of 1-ε. The above process is repeated until the sensor nodes are allocated. The delay of the AUV traveling between two sensor nodes is calculated and defined as the edge delay. The order of the AUV visiting the multiple nodes it is responsible for is defined as an asymmetric TSP problem. The LKH algorithm is used to solve the problem. The edge delay matrix of the AUV sailing between all sensor nodes is calculated and input into the optimization allocation strategy. The GA algorithm is used to generate the initial strategy φ, and the deep reinforcement learning algorithm is used to iteratively optimize the strategy φ until the optimal solution of the objective function is obtained.

[0008] Furthermore, the underwater area is divided into multiple 3D grids, where sensor nodes and AUVs are randomly deployed at the grid centroids; each grid is regarded as a point in the graph, and the travel delay of the AUV between adjacent grids is used as the weight of the edge to obtain a 3D grid weighted graph.

[0009] Furthermore, suppose that: the X-axis and Y-axis directions in the XYZ three-dimensional coordinates are parallel to the horizontal plane, and the Z-axis direction is perpendicular to the horizontal plane; t is the time variable; the speed of the sensor node in the X-axis direction is The speed of the sensor node in the Y-axis direction is The sensor node has a coordinate of X on the X axis. m (t), the sensor node is on the Y axis with coordinate Y m (t), the sensor node is on the Z axis with coordinate Z m (t); m represents the sensor node number;

[0010] The drift velocity model of the sensor node is constructed as follows:

[0011]

[0012] Among them, k1, k2, k3, λ, μ′ are parameters closely related to ocean currents, and k4 and k5 are random variables. Assuming that the floating of the node in the Z-axis direction is mainly caused by underwater noise, the coordinate drift model of the sensor node is as follows:

[0013]

[0014] where τ t is a random noise variable that follows a normal distribution.

[0015] Furthermore, the AUV moves from grid A to the adjacent grid B. Since the grid is filled with real ocean currents, the speed of the AUV is the synthesis of the AUV still water speed and the ocean current speed. The AUV still water speed component model is constructed as follows:

[0016]

[0017] According to the above equation, we can find the actual speed of AUV in grid A on the three coordinate axes:

[0018]

[0019] According to the above equation, we can find the actual speed of AUV in grid B on the three coordinate axes:

[0020]

[0021] Finally, the actual speed of AUV in grid A is

[0022] Finally, the actual speed of AUV in grid B is

[0023] The time delay of the AUV from the center point a of grid A to the center point b of grid B is calculated as:

[0024]

[0025] In the above formula:

[0026] represents the X-axis still water velocity component of the AUV in grid A;

[0027] represents the Y-axis still water velocity component of the AUV in grid A;

[0028] represents the Z-axis hydrostatic velocity component of the AUV in grid A;

[0029] Represents the ocean current component in the X-axis direction in the A grid;

[0030] Represents the ocean current component in the Y-axis direction in the A grid;

[0031] Represents the ocean current component in the Z-axis direction in the A grid;

[0032] represents the X-axis still water velocity component of the AUV in grid B;

[0033] represents the Y-axis still water velocity component of the AUV in grid B;

[0034] represents the Z-axis hydrostatic velocity component of the AUV in grid B;

[0035] represents the ocean current component in the X-axis direction in the B grid;

[0036] Represents the ocean current component in the Y-axis direction in the B grid;

[0037] Represents the ocean current component in the Z-axis direction in the B grid;

[0038] V represents the still water speed of the AUV;

[0039] No r() represents the standardization operation function;

[0040] o represents the center point of the contact surface between mesh A and mesh B;

[0041] a represents the center point of grid A;

[0042] b represents the center point of grid B;

[0043] dc aorepresents the predetermined direction vector of the AUV traveling from point a to point o;

[0044] t a,b represents the delay from grid A to grid B;

[0045] dis ao Indicates the distance from point a to point o;

[0046] dis ob Represents the distance from point o to point b;

[0047] represents the actual speed of the AUV in grid A;

[0048] Indicates the actual speed of the AUV in grid B.

[0049] Furthermore, let the grid α u and grid β u For AUV in a pair of sensor nodes n i and n j The uth pair of adjacent grids passed by when driving; Represents the grid α u To Grid β u The delay of n i The first sensor node number in the pair of sensor nodes; n j Number the second sensor node in the pair of sensor nodes;

[0050] Define the AUV as a pair of sensor nodes n i and n j The travel delay between sensor nodes n i and n j The edge delay weight between sensor nodes n i and n j The calculation formula of the edge delay weight between is as follows:

[0051]

[0052] Where:

[0053] delay i,j represents sensor node n i and n j The edge delay weight between them.

[0054] Furthermore, the method comprises the following steps:

[0055] Step 1: Based on A * The algorithm is used to optimize the AUV's driving trajectory between two different grids where two sensor nodes are located; let open setIndicates the grid where the AUV is currently located The set of all neighbor grids G nei Exclude the grid set formed after the grid has been passed; close set Represents the set of grids that the AUV has passed through before;

[0056] A * The algorithm will choose open set The grid with the smallest delay cost is taken as the grid to be passed later, and this grid is taken as the grid where the new AUV is currently located; the above process is repeated until the AUV reaches the grid where the target sensor node is located; once it reaches the target sensor node, close set All grids that the AUV passes through during its trajectory will be saved; then the edge delay between two sensor nodes is calculated based on the trajectory; using A * The algorithm calculates the edge delays between all sensor nodes and finally obtains the edge delay matrix between all sensor nodes;

[0057] Step 2: Assume that each AUV collects a set of sensor nodes; Assume that a set of sensor nodes is assigned to the AUV, and use the LKH algorithm to find the access path; Abstract the assigned sensor nodes as vertices in the graph, and abstract the edge delay of the AUV navigation trajectory between two sensor nodes as the weight of the edge between two vertices; The LKH algorithm takes this graph as input; Generate an initial feasible path based on this input, and set tour as the initial feasible path; Use the LKH algorithm to find the shortest travel path for each AUV, and finally obtain the path delay corresponding to each AUV;

[0058] Step 3: Based on the edge delay matrix between all sensor nodes and the path delay corresponding to each AUV; construct an optimized allocation strategy based on the GA algorithm and the N-step DQN algorithm; use the GA algorithm to generate the initial allocation strategy, and N-stepDQN performs secondary redistribution based on the initial allocation strategy; the N-step DQN algorithm will start from the first sensor node and allocate sensor nodes according to the optimized allocation strategy, and the sensor nodes that have not yet been allocated will be allocated according to the initial feasible strategy generated by the GA algorithm; therefore, the estimated path delay of task allocation can be calculated, and the reward of environmental feedback is no longer 0. When selecting actions, the ε greedy strategy is executed, and the sensor node is allocated to a certain AUV based on the current optimal task allocation strategy φ with a probability of 1-ε, and randomly allocated to a certain AUV with a probability of ε;

[0059] Then, it is determined whether all sensor nodes have been allocated. If so, the termination state is reached, otherwise, the termination state is not reached. Different methods are used to calculate the target value according to the termination state and the non-termination state.

[0060] The optimization allocation strategy algorithm continuously learns and eventually obtains the optimal task allocation strategy.

[0061] Furthermore, step 1 includes the following method steps:

[0062] Assume that the grid where the AUV is currently located is The grid to be selected is F s Representation Grid Delay cost; AUV needs to The adjacent grid set G nei Select a grid to form a set of grids to be selected open set ; Evaluate open set Each grid The delay cost F s , and select open set The one with the smallest delay cost is selected as the next grid, and Update it to it; at the same time, the previous Add to close set Medium; F s The AUV moves from the grid where the source sensor node is located to the grid The actual delay cost g s and from the grid The heuristic delay estimate h to the grid where the target sensor node is located s Composition; actual delay g s The calculation is expressed by the following formula:

[0063] g s =g c +t c , s ;

[0064] Where:

[0065] t c,s Represents the grid To Grid The time delay between

[0066] g c Indicates that the AUV moves from the grid where the source sensor node is located to the grid The actual delay of

[0067] According to AUV The trajectory to the grid where the target sensor node is located is calculated in turn along the three coordinate axes to calculate the heuristic delay evaluation h s ; This trajectory is divided into three sections; the first section is from To the grid with the same X-axis coordinate as the target sensor node The second paragraph is from Along the Y-axis direction, go to the grid with the same Y-axis coordinate as the grid where the target sensor node is located. The last paragraph is from Along the Z axis to the grid where the target sensor node is located; assuming that the grid sets that the AUV passes through in these three segments are and Where Ω x is the difference between the coordinate values ​​of the selected grid and the target grid on the X-axis; Ω y is the difference between the coordinate values ​​of the selected grid and the target grid on the Y axis; Ω z is the difference between the coordinate values ​​of the selected grid and the target grid on the Z axis;

[0068] The process is described as follows:

[0069]

[0070] Where:

[0071] u represents the set G x The grid number in ;

[0072] v indicates the set G y The grid number in ;

[0073] w represents the set G z The grid number in ;

[0074] t u,u+1 Represents AUV from the set G x The time delay from the uth grid to the u+1th grid;

[0075] t v,v+1 Represents AUV from the set G y The time delay from the vth grid to the v+1th grid;

[0076] t w,w+1 Represents AUV from the set G z The time delay from the wth grid to the w+1th grid;

[0077] The final delay is:

[0078] F s =g s +h s . .

[0079] Furthermore, step 2 includes the following method steps:

[0080] Assume n1, n2, n3 and n4 are vertices in tour; randomly select the initial vertex n1; then, select an edge x1=(n1,n2) in tour as the edge to be removed, and the corresponding edge y1=(n2,n3) not in tour as the connecting edge, and select x2=(n3,n4) in tour as the second edge to be removed, corresponding to y2=(n4,n1) as the second connecting edge; randomly select two edges in the iteration to remove them and replace them with two new edges; generate a new path tour′; if the path delay of tour′ is less than the path delay of tour, it is considered that the bilateral exchange is completed; otherwise, try a three-edge exchange; repeat the above steps until the path delay of tour′ is less than the path delay of tour or the number of exchanged edges is equal to 5; if the algorithm fails to find a better solution when reaching the five-edge exchange, it will select a new initial vertex in the loop and repeat the above steps; continue until every vertex in tour is used as the initial vertex, and finally output the optimal path.

[0081] Furthermore, step 3 includes the following method steps:

[0082] Formulate the following ε-greedy strategy:

[0083]

[0084] Where:

[0085] j is the sensor node number;

[0086] π(e|s) represents the ε-greedy action selection strategy;

[0087] |M| represents the number of selectable actions, and its value is equal to the number of AUVs;

[0088] e * is the action selected according to the optimal allocation strategy;

[0089] Represents the estimated Q value calculated based on the current Q network;

[0090] s j Indicates the state where the first j sensor nodes have been assigned;

[0091] A represents the state s j The set of optional actions in , i.e., all AUVs to which the j+1th sensor node can be assigned;

[0092] e represents the current state s j The action selected; i.e., the AUV to which the j+1th sensor node is assigned;

[0093] Indicates jThe corresponding state vector;

[0094] Indicates j+1 The corresponding state vector;

[0095] θ represents the parameters of the neural network;

[0096] In order to make the algorithm converge, ε decreases and tends to 0 as the algorithm iterates;

[0097] Execute action e to assign the corresponding sensor node to the selected AUV and obtain the corresponding state vector φ(s j+1 ) after j will move to the new state s j+1 ;s j+1 Indicates the state where the first j+1 sensor nodes have been assigned; φ(s j+1 ) means s j+1 The corresponding state vector;

[0098] Then, determine whether all sensor nodes have been assigned. If so, the terminal state is reached, otherwise the terminal state is not reached. Different methods are used to calculate the target Q value y according to the terminal state and non-terminal state. j ; The specific calculation formula is as follows:

[0099]

[0100] Where:

[0101] Among them, done is used to indicate s j+1 Whether it is terminated, that is, whether all sensors have been allocated;

[0102] r j Indicates the state is s j The reward of environmental feedback is determined by the path delay of the first j sensor nodes assigned;

[0103] Indicates that in state s j The target Q value at ;

[0104] r j (n) Indicates that in state s j The next n sensor nodes are allocated according to the strategy φ, and n path delays are obtained. These delays are accumulated as state s j Rewards;

[0105] γ represents the discount factor;

[0106] γ (n) represents the nth power of γ;

[0107] γ(k) represents γ to the power of k;

[0108] n and k represent the number of steps in N-step DQN;

[0109] r j+k+1 Indicates state s j+k+1 Rewards;

[0110] s j+n Indicates state s j The state after the nth step;

[0111] e′ represents the action of the target Q network;

[0112] θ′ represents the parameters of the target Q network;

[0113] Indicates that the target Q network state is s j+n , Q value when the action is e′;

[0114] Indicates that the target Q network is in state s j+n When , choose the action with the largest Q value;

[0115] Indicates state s j+n The corresponding state vector;

[0116] Assume that the n sensor nodes are assigned only for pre-evaluation rewards, but only the j+1th sensor node is actually assigned and the state is only updated to s j+1 ;at last, It is stored as a single sample in the experience replay buffer D. When there are enough samples in D, samples are extracted from D for loss calculation. The loss function is written as:

[0117]

[0118] Where:

[0119] L i (θ i ) represents the loss of network parameters at the i-th iteration;

[0120] It means to extract multiple samples from D and find the average value;

[0121] r represents the portion of reward elements in a single sample in D;

[0122] U(D) means uniform sampling;

[0123] θ i represents the network parameter value at the i-th iteration, where i represents the number of iterations;

[0124] θ′ i represents the network parameters of the target Q network at the i-th iteration;

[0125] Indicates that the target Q network is in state s j+n The Q value obtained by taking action e′ when

[0126] Indicates that the target Q network is in state s j+n When , choose the action with the largest Q value;

[0127] Indicates that the current Q network is in state s j+n The Q value obtained by taking action e when

[0128] Target network parameter θ′ i Only Q network parameters θ are used every C steps i is updated and is fixed between updates; it updates all parameters of the current Q network θ by backpropagating the gradient of the neural network as follows:

[0129]

[0130] Where:

[0131] Denotes the loss function L i For the parameter θ i The gradient of

[0132] Indicates that the current Q network is in state s j+n The Q value obtained by taking action e when

[0133] Denotes the action value function Q with respect to parameter θ i The gradient of

[0134] The optimization allocation strategy algorithm continuously learns and eventually obtains the optimal task allocation strategy.

[0135] The present invention also provides a device for an underwater multi-AUV task allocation method based on deep reinforcement learning, comprising a memory and a processor, wherein the memory is used to store a computer program; the processor is used to execute the computer program and implement the steps of the underwater multi-AUV task allocation method based on deep reinforcement learning as described above when executing the computer program.

[0136] The advantages and positive effects of the present invention are:

[0137] 1. Based on deep reinforcement learning, this paper designs an underwater multi-AUV task allocation algorithm suitable for ocean current scenarios. The allocation strategy of multiple AUVs can be optimized to reduce the delay when multiple AUVs collect data.

[0138] 2. The present invention grids the underwater scene in the system modeling stage and fills it with real ocean current data. The navigation problem of AUV in the ocean current environment is modeled, and the problem is abstracted into the minimum weight sum problem in a weighted directed graph according to the model. The straight-line distance used as the basis for task allocation in previous studies is converted into edge delay.

[0139] 3. The multi-AUV task allocation algorithm based on deep reinforcement learning designed in the algorithm design phase can optimize the task allocation strategy by continuously iteratively learning the internal relationship between sensor nodes. By introducing the navigation mode of AUV in the ocean current environment and combining it with our algorithm, the collection delay of multiple AUVs is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0140] Figure 1 It is a diagram of the underwater wireless sensor network multi-AUV data collection model.

[0141] Figure 2 This is a diagram illustrating the modeling method of AUV navigation between grids.

[0142] Figure 3 This is a workflow diagram of an underwater multi-AUV task allocation method based on deep reinforcement learning of the present invention.

[0143] In the figure:

[0144] o represents the center point of the contact surface between mesh A and mesh B.

[0145] a represents the center point of grid A.

[0146] b represents the center point of grid B.

[0147] c represents the center point of the grid C.

[0148] d represents the center point of the grid D.

[0149] o1 represents the center point of the contact surface between mesh A and mesh C.

[0150] o2 represents the center point of the contact surface between mesh B and mesh D.

[0151] GA-NDQN represents a combination of the GA algorithm and the N-step DQN algorithm.

[0152] dc ao represents the predetermined direction vector of the AUV traveling from point a to point o. DETAILED DESCRIPTION

[0153] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0154] The Chinese meanings of the following English words, phrases or abbreviations are as follows:

[0155] AUV: autonomous underwater vehicle.

[0156] UWSN: Underwater Wireless Sensor Networks, underwater wireless sensor networks.

[0157] Sink: sink node.

[0158] A* algorithm: is a path planning algorithm widely used in graph search. A* search algorithm is commonly known as A star algorithm. A* algorithm is one of the more popular heuristic search algorithms and is widely used in the field of path optimization. Its uniqueness is that it introduces global information when checking each possible node in the shortest path, estimates the distance from the current node to the end point, and uses it as a measure to evaluate the possibility that the node is on the shortest path.

[0159] GA algorithm: Genetic Algorithm (GA) was first proposed by John Holland of the United States in the 1970s. The algorithm was designed and proposed based on the laws of evolution of organisms in nature. It is a computational model of biological evolution that simulates the natural selection and genetic mechanism of Darwin's theory of biological evolution. It is a method of searching for the optimal solution by simulating the natural evolution process. The algorithm uses mathematical methods and computer simulation operations to convert the problem-solving process into a process similar to the crossover and mutation of chromosome genes in biological evolution. When solving more complex combinatorial optimization problems, it can usually obtain better optimization results faster than some conventional optimization algorithms.

[0160] N-step DQN algorithm: DQN (Deep Q-Network) is an algorithm based on deep learning and reinforcement learning, proposed by DeepMind, and is used to solve the Markov decision process (MDP) problem in discrete action space. The N-step DQN algorithm is a multi-step DQN algorithm. Multi-step DQN is a reinforcement learning algorithm that accelerates the learning process by considering the future n-step rewards, improves sample efficiency and training stability, and is particularly suitable for delayed reward environments.

[0161] ε Greedy Strategy: is a strategy used to solve the multi-armed bandit problem or Reinforcement Learning A strategy of exploring and exploiting trade-offs.

[0162] LKH algorithm: LKH (Lin-Kernighan-Helsgaun) algorithm is a heuristic algorithm for solving the traveling salesman problem (TSP). The core idea of ​​the algorithm is to gradually approach the optimal solution by iteratively optimizing the current solution. The LKH algorithm was proposed by Lin and Kernighan in 1973. Its characteristics are that it can find a better solution and has a certain degree of robustness.

[0163] Q-Network: Q-Network is a reinforcement learning method used to learn the expected reward for performing an action in a given state. DeepMind proposed the Deep Q-Network (DQN), which uses a deep neural network to approximate the Q function, so that it can learn the optimal strategy in a complex environment.

[0164] See also Figures 1 to 3 , an underwater multi-AUV task allocation method based on deep reinforcement learning, the underwater area is gridded and filled with real ocean current data; the AUV is assumed to sail between the grids; an optimization allocation strategy based on the deep reinforcement learning algorithm is established to allocate tasks to multiple AUVs, and the sensor nodes that each AUV needs to collect data and the access path along the sensor nodes are determined; the objective function is set to minimize the path delay for the completion of the total task, and the current optimal task allocation strategy is set as strategy φ; starting from the first sensor node, the sensor nodes are randomly allocated with a probability of ε, and the sensor nodes are allocated according to strategy φ with a probability of 1-ε; the above process is repeated until the sensor nodes are allocated; the delay of the AUV traveling between two sensor nodes is calculated and defined as the edge delay; the order of the AUV visiting the multiple nodes it is responsible for is defined as an asymmetric TSP problem; the LKH algorithm is used to solve the problem; the edge delay matrix of the AUV sailing between all sensor nodes is calculated and input into the optimization allocation strategy; the GA algorithm is used to generate the initial strategy φ, and the deep reinforcement learning algorithm is used to iteratively optimize the strategy φ until the optimal solution of the objective function is obtained.

[0165] Preferably, the underwater area can be divided into multiple 3D grids, where sensor nodes and AUVs are randomly deployed at the grid centroids; each grid is regarded as a point in the graph, and the travel delay of the AUV between adjacent grids is used as the weight of the edge to obtain a 3D grid weighted graph.

[0166] Preferably, it can be assumed that: the X-axis and Y-axis directions in the XYZ three-dimensional coordinates are parallel to the horizontal plane, and the Z-axis direction is perpendicular to the horizontal plane; t is the time variable; the speed of the sensor node in the X-axis direction is The speed of the sensor node in the Y-axis direction is The sensor node has a coordinate of X on the X axis.m (t), the sensor node is on the Y axis with coordinate Y m (t), the sensor node is on the Z axis with coordinate Z m (t); m represents the sensor node number;

[0167] The drift velocity model of the sensor node is constructed as follows:

[0168]

[0169] Among them, k1, k2, k3, λ, μ′ are parameters closely related to ocean currents, and k4 and k5 are random variables. It can be assumed that the floating of the node in the Z-axis direction is mainly caused by underwater noise. The coordinate drift model of the sensor node is as follows:

[0170]

[0171] where τ t is a random noise variable that follows a normal distribution.

[0172] Preferably, the AUV travels from grid A to adjacent grid B. Since the grid is filled with real ocean currents, the speed of the AUV is the synthesis of the AUV still water speed and the ocean current speed. The AUV still water speed component model can be constructed as follows:

[0173]

[0174] According to the above equation, we can find the actual speed of AUV in grid A on the three coordinate axes:

[0175]

[0176] Similarly, according to the above equation, the components of the actual speed of the AUV in grid B on the three coordinate axes can be obtained:

[0177]

[0178] Finally, the actual speed of AUV in grid A is

[0179] Finally, the actual speed of AUV in grid B is The time delay of the AUV from the center point a of grid A to the center point b of grid B is calculated as:

[0180]

[0181] In the above formula:

[0182] represents the X-axis still water velocity component of the AUV in grid A;

[0183] represents the Y-axis still water velocity component of the AUV in grid A;

[0184] represents the Z-axis hydrostatic velocity component of the AUV in grid A;

[0185] Represents the ocean current component in the X-axis direction in the A grid;

[0186] Represents the ocean current component in the Y-axis direction in the A grid;

[0187] Represents the ocean current component in the Z-axis direction in the A grid;

[0188] represents the X-axis still water velocity component of the AUV in grid B;

[0189] represents the Y-axis still water velocity component of the AUV in grid B;

[0190] represents the Z-axis hydrostatic velocity component of the AUV in grid B;

[0191] represents the ocean current component in the X-axis direction in the B grid;

[0192] Represents the ocean current component in the Y-axis direction in the B grid;

[0193] Represents the ocean current component in the Z-axis direction in the B grid;

[0194] V represents the still water speed of the AUV;

[0195] No r() represents the standardization operation function;

[0196] o represents the center point of the contact surface between mesh A and mesh B;

[0197] a represents the center point of grid A;

[0198] b represents the center point of grid B;

[0199] dc ao represents the predetermined direction vector of the AUV traveling from point a to point o;

[0200] t a,b represents the delay from grid A to grid B;

[0201] dis ao Indicates the distance from point a to point o;

[0202] dis ob Represents the distance from point o to point b;

[0203] represents the actual speed of the AUV in grid A;

[0204] Indicates the actual speed of the AUV in grid B.

[0205] Preferably, the grid α can be set u and grid β u For AUV in a pair of sensor nodes n i and n j The uth pair of adjacent grids passed by when driving; Represents the grid α u To Grid β u The delay of n i The first sensor node number in the pair of sensor nodes; n j Number the second sensor node in the pair of sensor nodes;

[0206] The AUV can be defined as a pair of sensor nodes n i and n j The travel delay between sensor nodes n i and n j The edge delay weight between sensor nodes n i and n j The calculation formula of the edge delay weight between is as follows:

[0207]

[0208] Where:

[0209] delay i,j represents sensor node n i and n j The edge delay weight between them.

[0210] Preferably, the method may include the following steps:

[0211] Step 1: Based on A * The algorithm is used to optimize the AUV's driving trajectory between two different grids where two sensor nodes are located; let open set Indicates the grid where the AUV is currently located The set of all neighbor grids G nei Exclude the grid set formed after the grid has been passed; close set Represents the set of grids that the AUV has passed through before;

[0212] A *The algorithm will choose open set The grid with the smallest delay cost is taken as the grid to be passed later, and this grid is taken as the grid where the new AUV is currently located; the above process is repeated until the AUV reaches the grid where the target sensor node is located; once it reaches the target sensor node, close set All grids that the AUV passes through during its trajectory will be saved; then the edge delay between two sensor nodes is calculated based on the trajectory; using A * The algorithm calculates the edge delays between all sensor nodes and finally obtains the edge delay matrix between all sensor nodes;

[0213] Step 2: Each AUV can collect a set of sensor nodes; Assume that a set of sensor nodes is assigned to the AUV, and use the LKH algorithm to find the access path; The assigned sensor nodes are abstracted as vertices in the graph, and the edge delay of the AUV navigation trajectory between two sensor nodes is abstracted as the weight of the edge between the two vertices; The LKH algorithm takes this graph as input; Based on this input, an initial feasible path is generated, and tour is set as the initial feasible path; Use the LKH algorithm to find the shortest travel path for each AUV, and finally obtain the path delay corresponding to each AUV;

[0214] Step 3: Based on the edge delay matrix between all sensor nodes and the path delay corresponding to each AUV; an optimized allocation strategy can be constructed based on the GA algorithm and the N-step DQN algorithm; the GA algorithm can be used to generate an initial allocation strategy, and the N-step DQN algorithm performs secondary redistribution based on the initial allocation strategy; the N-step DQN algorithm can be used to allocate sensor nodes starting from the first sensor node according to the optimized allocation strategy, and the sensor nodes that have not yet been allocated will be allocated according to the initial feasible strategy generated by the GA algorithm; therefore, the estimated path delay of the task allocation can be calculated, and the reward of the environmental feedback is no longer 0. When selecting an action, the ε greedy strategy is executed, and the sensor node is allocated to a certain AUV based on the current optimal task allocation strategy φ with a probability of 1-ε, and randomly allocated to a certain AUV with a probability of ε;

[0215] Then, it is determined whether all sensor nodes have been allocated. If so, the termination state is reached, otherwise, the termination state is not reached. Different methods are used to calculate the target value according to the termination state and the non-termination state.

[0216] The optimization allocation strategy algorithm continuously learns and eventually obtains the optimal task allocation strategy.

[0217] Preferably, step 1 may include the following method steps:

[0218] Assume that the grid where the AUV is currently located is The grid to be selected is F s Representation Grid Delay cost; AUV needs to The adjacent grid set G nei Select a grid to form a set of grids to be selected open set ; Evaluate open set Each grid The delay cost F s , and select open set The one with the smallest delay cost is selected as the next grid, and Update it to it; at the same time, the previous Add to close set Medium; F s The AUV moves from the grid where the source sensor node is located to the grid The actual delay cost g s and from the grid The heuristic delay estimate h to the grid where the target sensor node is located s Composition; actual delay g s The calculation is expressed by the following formula:

[0219] g s =g c +t c,s ;

[0220] Where:

[0221] t c,s Represents the grid To Grid The time delay between

[0222] g c Indicates that the AUV moves from the grid where the source sensor node is located to the grid The actual delay of

[0223] According to the AUV The trajectory to the grid where the target sensor node is located is calculated in turn along the three coordinate axes to calculate the heuristic delay evaluation h s ; This trajectory is divided into three sections; the first section is from To the grid with the same X-axis coordinate as the target sensor node The second paragraph is from Along the Y-axis direction, go to the grid with the same Y-axis coordinate as the grid where the target sensor node is located. The last paragraph is from Along the Z axis to the grid where the target sensor node is located; assuming that the grid sets that the AUV passes through in these three segments are and Where Ω xis the difference between the coordinate values ​​of the selected grid and the target grid on the X-axis; Ω y is the difference between the coordinate values ​​of the selected grid and the target grid on the Y axis; Ω z is the difference between the coordinate values ​​of the selected grid and the target grid on the Z axis;

[0224] The process can be described as follows:

[0225]

[0226] Where:

[0227] u represents the set G x The grid number in ;

[0228] v indicates the set G y The grid number in ;

[0229] w represents the set G z The grid number in ;

[0230] t u,u+1 Represents AUV from the set G x The time delay from the uth grid to the u+1th grid;

[0231] t v,v+1 Represents AUV from the set G y The time delay from the vth grid to the v+1th grid;

[0232] t w,w+1 Represents AUV from the set G z The time delay from the wth grid to the w+1th grid;

[0233] The final delay is:

[0234] F s =g s +h s . .

[0235] Preferably, step 2 comprises the following method steps:

[0236] Assume that n1, n2, n3 and n4 are vertices in tour; randomly select the initial vertex n1; then, select an edge x1=(n1,n2) in tour as the edge to be removed, and the corresponding edge y1=(n2,n3) not in tour as the connecting edge, and select x2=(n3,n4) in tour as the second edge to be removed, corresponding to y2=(n4,n1) as the second connecting edge; randomly select two edges in the iteration to remove them and replace them with two new edges; generate a new path tour′; if the path delay of tour′ is less than the path delay of tour, it is considered that the bilateral exchange is completed; otherwise, try a three-edge exchange; repeat the above steps until the path delay of tour′ is less than the path delay of tour or the number of exchanged edges is equal to 5; if the algorithm fails to find a better solution when reaching the five-edge exchange, it will select a new initial vertex in the loop and repeat the above steps; execute until every vertex in tour is used as the initial vertex, and finally output the optimal path.

[0237] Preferably, step 3 may include the following method steps:

[0238] Formulate the following ε-greedy strategy:

[0239]

[0240] Where:

[0241] j is the sensor node number;

[0242] π(e|s) represents the ε-greedy action selection strategy;

[0243] |M| represents the number of selectable actions, and its value is equal to the number of AUVs;

[0244] e * is the action selected according to the optimal allocation strategy;

[0245] Represents the estimated Q value calculated based on the current Q network;

[0246] s j Indicates the state where the first j sensor nodes have been assigned;

[0247] A represents the state s j The set of optional actions in , i.e., all AUVs to which the j+1th sensor node can be assigned;

[0248] e represents the current state s j The action selected; i.e., the AUV to which the j+1th sensor node is assigned;

[0249] Indicates jThe corresponding state vector;

[0250] Indicates j+1 The corresponding state vector;

[0251] θ represents the parameters of the neural network;

[0252] In order to make the algorithm converge, ε decreases and tends to 0 as the algorithm iterates;

[0253] Execute action e to assign the corresponding sensor node to the selected AUV and obtain the corresponding state vector φ(s j+1 ) after j will move to the new state s j+1 ;s j+1 Indicates the state where the first j+1 sensor nodes have been assigned; φ(s j+1 ) means s j+1 The corresponding state vector;

[0254] Then, determine whether all sensor nodes have been assigned. If so, the terminal state is reached, otherwise the terminal state is not reached. Different methods are used to calculate the target Q value y according to the terminal state and non-terminal state. j ; The specific calculation formula is as follows:

[0255]

[0256] Where:

[0257] Among them, done is used to indicate s j+1 Whether it is terminated, that is, whether all sensors have been allocated;

[0258] r j Indicates the state is s j The reward of environmental feedback is determined by the path delay of the first j sensor nodes assigned;

[0259] Indicates that in state s j The target Q value at ;

[0260] r j (n) Indicates that in state s j The next n sensor nodes are allocated according to the strategy φ, and n path delays are obtained. These delays are accumulated as state s j Rewards;

[0261] γ represents the discount factor;

[0262] γ (n) represents the nth power of γ;

[0263] γ(k) represents γ to the power of k;

[0264] n and k represent the number of steps in N-step DQN;

[0265] r j+k+1 Indicates state s j+k+1 Rewards;

[0266] s j+n Indicates state s j The state after the nth step;

[0267] e′ represents the action of the target Q network;

[0268] θ′ represents the parameters of the target Q network;

[0269] Indicates that the target Q network state is s j+n , Q value when the action is e′;

[0270] Indicates that the target Q network is in state s j+n When , choose the action with the largest Q value;

[0271] Indicates state s j+n The corresponding state vector;

[0272] Assume that the n sensor nodes are assigned only for pre-evaluation rewards, but only the j+1th sensor node is actually assigned and the state is only updated to s j+1 ;at last, It is stored as a single sample in the experience replay buffer D. When there are enough samples in D, samples are extracted from D for loss calculation. The loss function is written as:

[0273]

[0274] Where:

[0275] L i (θ i ) represents the loss of network parameters at the i-th iteration;

[0276] It means to extract multiple samples from D and find the average value;

[0277] r represents the portion of reward elements in a single sample in D;

[0278] U(D) means uniform sampling;

[0279] θ i represents the network parameter value at the i-th iteration, where i represents the number of iterations;

[0280] θ′ i represents the network parameters of the target Q network at the i-th iteration;

[0281] Indicates that the target Q network is in state s j+n The Q value obtained by taking action e′ when

[0282] Indicates that the target Q network is in state s j+n When , choose the action with the largest Q value;

[0283] Indicates that the current Q network is in state s j+n The Q value obtained by taking action e when

[0284] Target network parameter θ′ i Only Q network parameters θ are used every C steps i is updated and is fixed between updates; it updates all parameters of the current Q network θ by backpropagating the gradient of the neural network as follows:

[0285]

[0286] Where:

[0287] Denotes the loss function L i For the parameter θ i The gradient of

[0288] Indicates that the current Q network is in state s j+n The Q value obtained by taking action e when

[0289] Denotes the action value function Q with respect to parameter θ i The gradient of

[0290] The optimization allocation strategy algorithm continuously learns and eventually obtains the optimal task allocation strategy.

[0291] The present invention also provides a device for an underwater multi-AUV task allocation method based on deep reinforcement learning, comprising a memory and a processor, wherein the memory is used to store a computer program; the processor is used to execute the computer program and implement the steps of the underwater multi-AUV task allocation method based on deep reinforcement learning as described above when executing the computer program.

[0292] The following is a preferred embodiment of the present invention to further illustrate the working process and working principle of the present invention:

[0293] To describe the implementation method more clearly, it is assumed that a UWSN consists of a base station, a sink node, a sensor node set N and an AUV set M. The base station is deployed on land, the sink node is deployed on the water surface, and the sensor node n i ∈N and AUVm k ∈M are randomly deployed underwater. Sensor nodes n i Collect ocean data to generate data packets, and the AUV receives node data within the communication range through acoustic communication, and finally the AUV transmits the data packets to the sink node. For each AUV, they need to start from the sink node, collect the node data assigned to them in turn, and finally return to the sink node. The specific method, structure, characteristics and function of the underwater multi-AUV task allocation algorithm designed according to the present invention are described in detail below in conjunction with the accompanying drawings.

[0294] 1. System modeling phase

[0295] Step A1: Due to ocean currents, sensor nodes will move in the ocean. In addition, ocean currents are mainly affected by horizontal wind force, so the present invention mainly considers the moving speed of the node in the horizontal direction and ignores the speed in the vertical direction.

[0296] Assume that: the X-axis and Y-axis in the XYZ three-dimensional coordinates are parallel to the horizontal plane, and the Z-axis is perpendicular to the horizontal plane; t is the time variable; the speed of the sensor node in the X-axis direction is The speed of the sensor node in the Y-axis direction is The sensor node has a coordinate of X on the X axis. m (t), the sensor node is on the Y axis with coordinate Y m (t), the sensor node is on the Z axis with coordinate Z m (t); m represents the sensor node number.

[0297] In order to enable the AUV to predict the location of the sensor node and continuously collect data, the sensor node drift velocity model is modeled as follows:

[0298]

[0299] Among them, k1, k2, k3, λ, μ′ are parameters closely related to ocean currents, and k4 and k5 are random variables. Based on the analysis in the previous paragraph and assuming that the floating of the node in the Z-axis direction is mainly caused by underwater noise, the coordinate drift model of the sensor node is as follows:

[0300]

[0301] where τ t is a random noise variable that follows a normal distribution.

[0302] Step A2: We divide the 3D underwater scene into grids, where sensor nodes and AUVs are randomly deployed at the grid centroids. If each grid is regarded as a point in the graph and the travel delay of AUVs between adjacent grids is regarded as the weight of the edge, it is a weighted graph. To fit the actual situation, the grids are filled with ocean currents generated by real data, such as Figure 2 As shown. Points a and b in the figure are the centers of the two grids A and B respectively, and point o is the center of the contact surface of the two grids. Due to the direction of the water flow, the AUV may move back and forth between adjacent grids at different times, that is, the weights of the two directions of the edge in the weighted graph are different, thus forming a weighted directed graph. In other words, a→b has different weights compared to b→a. The weight is related to the AUV still water speed and the current speed. Let the still water speed of the AUV be a constant value V. Similarly, due to the water flow, the AUV needs to change the still water speed components in different directions so that when synthesized with the water flow speed, the AUV can move in a predetermined direction. The still water speed components of the AUV in different directions should satisfy the following equations:

[0303]

[0304] According to the above equation, we can find the actual speed of AUV in grid A on the three coordinate axes:

[0305]

[0306] Usually, the ocean current speeds in grid A and grid B are different. According to equation (3), the still water speed of the AUV is a constant value V, so the actual speed of the AUV will change with the change of the current speed. Therefore, the actual speed of the AUV in grid B is Need to recalculate.

[0307] Similarly, according to formula (3), the components of the actual speed of the AUV in grid B on the three coordinate axes can be obtained:

[0308]

[0309] Finally, the actual speed of AUV in grid A is

[0310] Finally, the actual speed of AUV in grid B is

[0311] Finally, the delay from the center point a of grid A to the center point b of grid B can be calculated as:

[0312]

[0313] In the above formula:

[0314] represents the X-axis still water velocity component of the AUV in grid A; represents the Y-axis still water velocity component of the AUV in grid A; represents the Z-axis hydrostatic velocity component of the AUV in grid A; Represents the ocean current component in the X-axis direction in the A grid;

[0315] Represents the ocean current component in the Y-axis direction in the A grid;

[0316] Represents the ocean current component in the Z-axis direction in the A grid;

[0317] represents the X-axis still water velocity component of the AUV in grid B; represents the Y-axis still water velocity component of the AUV in grid B; represents the Z-axis hydrostatic velocity component of the AUV in grid B; represents the ocean current component in the X-axis direction in the B grid;

[0318] Represents the ocean current component in the Y-axis direction in the B grid;

[0319] Represents the ocean current component in the Z-axis direction in the B grid;

[0320] V represents the still water speed of the AUV;

[0321] No r() represents the standardization operation function;

[0322] o represents the center point of the contact surface between mesh A and mesh B;

[0323] a represents the center point of grid A;

[0324] b represents the center point of grid B;

[0325] dc ao represents the predetermined direction vector of the AUV traveling from point a to point o;

[0326] t a,b represents the delay from grid A to grid B;

[0327] dis ao Indicates the distance from point a to point o;

[0328] dis ob Represents the distance from point o to point b;

[0329] represents the actual speed of the AUV in grid A;

[0330] Indicates the actual speed of the AUV in grid B.

[0331] An AUV starts from the grid where the current sensor node is located and may pass through several grids to reach the grid where another sensor node is located. In addition, the AUV in the grid can move to adjacent grids in different directions through grid edges, grid faces, or grid vertices. Figure 2 As shown in Figure 1, assuming that there is a sensor node in each grid C and D, the AUV starts from grid C, passes through grids A and B in turn, and reaches grid D. Different ways of passing through the grid edge mean that different delays are required for the AUV to reach the grid surface or grid vertex of its adjacent grid. Similar to formula (5), we can calculate the delay of the AUV between grid C and grid A, and between grid B and grid D as t c,a ,t b,d The total delay between grid C and grid D is tc ,a +t a,b +t b,d .t c,a represents the delay from grid C to grid A; t b,d Represents the delay from grid B to grid D.

[0332] Therefore, we give the AUV in a pair of sensor nodes n i and n j For a more general case of the delay between travels, let the grid α u and grid β u For AUV in a pair of sensor nodes n i and n j The uth pair of adjacent grids passed by when driving; Represents the grid α u To Grid β u The delay of n i The first sensor node number in the pair of sensor nodes; n j Number the second sensor node in the pair of sensor nodes;

[0333] Define the AUV as a pair of sensor nodes n i and n j The travel delay between sensor nodes n i and n j The edge delay weight between sensor nodes n i and n j The calculation formula of the edge delay weight between is as follows:

[0334]

[0335] Where:

[0336] delay i,j represents sensor node n i and n j The edge delay weight between them.

[0337] Step A3: The present invention focuses on determining the sensor nodes that each AUV needs to collect data from and the access paths along the sensor nodes through the task allocation of multiple AUVs. Assume that the task allocation strategy is Then AUVm k The set of sensor nodes responsible for collection is Indicates that it is assigned to AUVm k The number of sensor nodes for data collection. Correspondingly, the path that the AUV needs to travel can be expressed as in Indicates AUVm k The fth sensor node visited, so when the allocation strategy is When AUVm k The path delay is:

[0338]

[0339] delay f,f+1 Represents a sensor node The invention defines represents the overall path delay, which is the strategy The maximum value of all AUV path delays under , can be expressed as:

[0340]

[0341] The goal of the present invention is to find an optimal Make delay φ Minimum.

[0342] 2. Algorithm design stage

[0343] After building the system model, we need to find the optimal multi-AUV task allocation strategy under the current 3D grid model. Our algorithm can be divided into three parts: AUV edge delay calculation between sensor nodes, AUV path delay calculation along a set of sensor nodes, and AUV task allocation.

[0344] Step B1: In order to reduce the travel delay of AUV between any sensor nodes, a *The algorithm is used to optimize the AUV’s driving trajectory between the two different grids where the two sensor nodes are located. Specifically, the AUV starts from the grid where the source sensor node is located and uses A * The algorithm continuously searches for the next grid to pass through until it reaches the grid where the target sensor node is located. In this process, it is assumed that the grid where the AUV is currently located is The grid to be selected is F s Representation Grid delay cost; next, the AUV needs to The adjacent grid set G nei Select a grid to form a set of grids to be selected open set The reason why we don’t directly start from G nei The next grid is selected because G nei Some of the grids in have been passed by the AUV before, so we need to exclude these grids. set Each grid The delay cost F s , and select open set The one with the smallest delay cost is selected as the next grid, and Update it to it. At the same time, replace the previous Add to close set In. F s The AUV moves from the grid where the source sensor node is located to the grid The actual delay cost g s and from the grid The heuristic delay estimate h to the grid where the target sensor node is located s Composition. Actual delay g s The calculation is expressed by the following formula:

[0345] g s =g c +t c,s (9)

[0346] Where:

[0347] t c,s Represents the grid To Grid The time delay between

[0348] g c Indicates that the AUV moves from the grid where the source sensor node is located to the grid The actual delay.

[0349] Calculated from (5), it means that from the grid arrive of delay.

[0350] According to AUV The trajectory to the grid where the target sensor node is located calculates the heuristic delay evaluation h in the three coordinate axes in turn s It is worth noting that this trajectory is only used to calculate h s , the AUV does not actually travel along it. This trajectory can be divided into three segments. The first segment is from To the grid with the same X-axis coordinate as the target sensor node The second paragraph is from Along the Y-axis direction, go to the grid with the same Y-axis coordinate as the grid where the target sensor node is located. The last paragraph is from Along the Z axis to the grid where the target sensor node is located. Assume that the grid sets that the AUV passes through in these three segments are and Where Ω x ,Ω y ,Ω z are the coordinate values ​​of the candidate grid and the target grid on the X, Y, and Z axes respectively. The process is described as follows:

[0351]

[0352] Where:

[0353] u represents the set G x The grid number in ;

[0354] v indicates the set G y The grid number in ;

[0355] w represents the set G z The grid number in ;

[0356] t u,u+1 Represents AUV from the set G x The time delay from the uth grid to the u+1th grid;

[0357] t v,v+1 Represents AUV from the set G y The time delay from the vth grid to the v+1th grid;

[0358] t w,w+1 Represents AUV from the set G z The time delay from the wth grid to the w+1th grid;

[0359] Finally, the delay can be obtained as:

[0360] F s =g s +h s (11)

[0361] A * The algorithm will choose open set The grid with the smallest delay cost is taken as the grid to be passed later, and this grid is taken as the grid where the new AUV is currently located; the above process is repeated until the AUV reaches the grid where the target sensor node is located; once it reaches the target sensor node, close set All grids that the AUV passes through during its trajectory will be saved; then the edge delay between two sensor nodes is calculated based on the trajectory; using A * The algorithm calculates the edge delays between all sensor nodes and finally obtains the edge delay matrix between all sensor nodes;

[0362] Then, we can use formula (5) to calculate the edge delay between two sensor nodes based on the trajectory. * The algorithm can calculate the edge delay between all sensor nodes and finally obtain the edge delay matrix delay between all sensor nodes. matrix .

[0363] Step B2: Each AUV should collect a set of sensor nodes to efficiently complete the data collection task. Assuming that a set of sensor nodes is assigned to the AUV, the present invention uses the LKH algorithm to find the access path. We abstract the assigned sensor nodes as vertices in the graph, and abstract the edge delay of the AUV navigation trajectory between two sensor nodes as the weight of the edge between the two vertices. The LKH algorithm takes this graph as input. Based on this input, it generates an initial feasible path tour. Next, two edges are randomly selected in the iteration to remove them and replace them with two new edges. Specifically, let n1, n2, n3 and n4 be vertices in tour; the initial vertex n1 is randomly selected. Then, an edge x1=(n1,n2) in tour is selected as the edge to be removed, and an edge y1=(n2,n3) not in tour is selected as the connecting edge, and x2=(n3,n4) in tour is selected as the second removed edge, corresponding to y2=(n4,n1) as the second connecting edge. n1, n2, n3 and n4 are vertices in tour. The replacement will generate a new path tour′. If the path delay of tour′ is less than the path delay of tour, it means that a better path is obtained, which is called completing the bilateral exchange. Otherwise, try the three-edge exchange. In general, if the algorithm fails to find a better solution when it reaches the five-edge exchange, it will select a new initial vertex n1 in the loop and repeat the above two-edge exchange and three-edge exchange until the five-edge exchange. Continue until each vertex is used as the initial vertex, and finally output the optimal path. Use the LKH algorithm to find the shortest travel path for each AUV, and finally get the path delay corresponding to each AUV.

[0364] Step B3: Get the edge delay matrix delay matrix And it can calculate the path delay corresponding to each AUV. The present invention uses the deep reinforcement learning algorithm GA-NDQN to optimize the allocation strategy. The GA-NDQN optimization allocation strategy is a combination algorithm that combines the GA algorithm and the N-step DQN algorithm.

[0365] The multi-AUV task allocation problem has its particularity, that is, there is a delay only after the last sensor node is allocated, which is the sparse reward problem in reinforcement learning. Therefore, we introduce GA to generate an initial allocation strategy, and then we can use this strategy to complete the allocation of the remaining unallocated sensor nodes. Through such processing, the estimated path delay of the currently completed partial task allocation can be calculated, so that the reward of environmental feedback is no longer 0. NDQN will allocate sensor nodes according to the current optimal strategy φ starting from the first sensor node, and the sensor nodes that have not yet been allocated will be allocated according to the initial feasible strategy generated by GA. GA-NDQN will allocate the next sensor node with a probability of 1-ε based on the strategy φ, and the next sensor node will be randomly assigned to the AUV with a probability ε. This is the ε-greedy strategy, which will be executed when selecting an action. It can be expressed as:

[0366]

[0367] e * is the action selected according to the policy φ, which can be expressed as

[0368] Where:

[0369] j is the sensor node number;

[0370] π(e|s) represents the ε-greedy action selection strategy;

[0371] |M| represents the number of selectable actions, and its value is equal to the number of AUVs;

[0372] e * is the action selected according to the optimal allocation strategy;

[0373] Represents the estimated Q value calculated based on the current Q network;

[0374] s j Indicates the state where the first j sensor nodes have been assigned;

[0375] A represents the state s j The set of optional actions in , i.e., all AUVs to which the j+1th sensor node can be assigned;

[0376] e represents the current state s jThe action selected; i.e., the AUV to which the j+1th sensor node is assigned;

[0377] Indicates j The corresponding state vector;

[0378] Indicates j+1 The corresponding state vector;

[0379] θ represents the parameters of the neural network;

[0380] Specifically, A represents all AUVs to which the j+1th sensor node can be assigned, and e represents the AUV to which the sensor node is assigned. Indicates j The corresponding state vector can be obtained through operations such as dimensionality reduction. The state vector is more conducive to the learning of the neural network. In order to make the algorithm converge, ε decreases and tends to 0 as the algorithm iterates. Execute action e to assign the corresponding sensor node to the selected AUV and obtain the corresponding state vector After j will move to the new state s j+1 Then, GA-NDQN will determine whether all sensor nodes have been assigned, that is, whether the terminal state has been reached, and calculate the target Q value y using different methods according to the terminal state and non-terminal state. i The specific calculation formula is as follows:

[0381]

[0382] Where:

[0383] Among them, done is used to indicate s j+1 Whether it is terminated, that is, whether all sensors have been allocated;

[0384] r j Indicates the state is s j The reward of environmental feedback is determined by the path delay of the first j sensor nodes assigned;

[0385] Indicates that in state s j The target Q value at ;

[0386] r j (n) Indicates that in state s j The next n sensor nodes are allocated according to the strategy φ, and n path delays are obtained. These delays are accumulated as state s j Rewards;

[0387] γ represents the discount factor;

[0388] γ (n) represents the nth power of γ;

[0389] γ (k) represents γ to the power of k;

[0390] n and k represent the number of steps in N-step DQN;

[0391] r j+k+1 Indicates state s j+k+1 Rewards;

[0392] s j+n Indicates state s j The state after the nth step;

[0393] e′ represents the action of the target Q network;

[0394] θ′ represents the parameters of the target Q network;

[0395] Indicates that the target Q network state is s j+n , Q value when the action is e′;

[0396] Indicates that the target Q network is in state s j+n When , choose the action with the largest Q value;

[0397] Indicates state s j+n The corresponding state vector;

[0398] Assume that the n sensor nodes are assigned only for pre-evaluation rewards, but only the j+1th sensor node is actually assigned and the state is only updated to s j+1 .at last, It is stored as a single sample in the experience replay buffer D. In fact, a sample can be generated each time it is completed and placed in D. When there are enough samples in D, samples are extracted from D for loss calculation. The loss function can be written as:

[0399]

[0400] Where:

[0401] L i (θ i ) represents the loss of network parameters at the i-th iteration;

[0402] It means to extract multiple samples from D and find the average value;

[0403] r represents the portion of reward elements in a single sample in D;

[0404] U(D) means uniform sampling;

[0405] θ i represents the network parameter value at the i-th iteration, where i represents the number of iterations;

[0406] θ′ i represents the network parameters of the target Q network at the i-th iteration;

[0407] Indicates that the target Q network is in state s j+n The Q value obtained by taking action e′ when

[0408] Indicates that the target Q network is in state s j+n When , choose the action with the largest Q value;

[0409] Indicates that the current Q network is in state s j+n The Q value obtained by taking action e when

[0410] Target network parameter θ′ i Only Q network parameters θ are used every C steps i It updates all parameters of the current Q network θ by back propagating the gradient of the neural network as follows:

[0411]

[0412] Where:

[0413] Denotes the loss function L i For the parameter θ i The gradient of

[0414] Indicates that the current Q network is in state s j+n The Q value obtained by taking action e when

[0415] Denotes the action value function Q with respect to parameter θ i The gradient of

[0416] The GA-NDQN algorithm continuously learns and finally obtains the optimal task allocation strategy φ.

[0417] A * The algorithms such as the GA algorithm, the N-step DQN algorithm, the ε greedy strategy, the LKH algorithm, etc. can all adopt applicable algorithms and functional modules in the prior art, or adopt software, algorithms and functional modules in the prior art and be constructed by conventional technical means.

[0418] The embodiments described above are only used to illustrate the technical ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The patent scope of the present invention cannot be limited only by this embodiment, that is, any equivalent changes or modifications made to the spirit disclosed by the present invention still fall within the patent scope of the present invention.

Claims

1. An underwater multi-AUV task allocation method based on deep reinforcement learning, characterized in that: The underwater area is gridded and filled with real ocean current data; the AUV is set to sail between the grids; an optimization allocation strategy based on a deep reinforcement learning algorithm is established to allocate tasks to multiple AUVs, determine the sensor nodes that each AUV needs to collect data from and the access paths along the sensor nodes; Set the objective function to minimize the path delay of the total task completion, set the current optimal task allocation strategy as strategy φ; start from the first sensor node, randomly assign sensor nodes with a probability of ε, and assign sensor nodes according to strategy φ with a probability of 1-ε; Repeat the above process until the sensor nodes are allocated; Calculate the time delay of AUV traveling between two sensor nodes and define it as edge delay; define the order in which AUV visits multiple nodes as an asymmetric TSP problem; use LKH algorithm to solve the problem; The edge delay matrix of the AUV's navigation between all sensor nodes is calculated and input into the optimization allocation strategy. The GA algorithm is used to generate the initial strategy φ, and then the deep reinforcement learning algorithm is used to iteratively optimize the strategy φ until the optimal solution of the objective function is obtained.

2. The underwater multi-AUV task allocation method based on deep reinforcement learning according to claim 1 is characterized in that: The underwater area is divided into multiple 3D grids, where sensor nodes and AUVs are randomly deployed at the grid centroids. Each grid is regarded as a point in the graph, and the travel delay of AUV between adjacent grids is used as the weight of the edge to obtain a 3D grid weighted graph.

3. The underwater multi-AUV task allocation method based on deep reinforcement learning according to claim 1 is characterized in that: In the XYZ three-dimensional coordinate system, the X-axis and Y-axis directions are parallel to the horizontal plane, and the Z-axis direction is perpendicular to the horizontal plane; t is the time variable; the speed of the sensor node in the X-axis direction is The speed of the sensor node in the Y-axis direction is The sensor node has a coordinate of X on the X axis. m (t), the sensor node is on the Y axis with coordinate Y m (t), the sensor node is on the Z axis with coordinate Z m (t); m represents the sensor node number; The drift velocity model of the sensor node is constructed as follows: where k1, k2, k3, λ, μ ′ is a parameter closely related to ocean currents, k4 and k5 are random variables. Assuming that the floating of the node in the Z-axis direction is mainly caused by underwater noise, the coordinate drift model of the sensor node is as follows: where τ t is a random noise variable that follows a normal distribution.

4. The underwater multi-AUV task allocation method based on deep reinforcement learning according to claim 3 is characterized in that: The AUV travels from grid A to adjacent grid B. Since the grid is filled with real ocean currents, the speed of the AUV is the combination of the AUV still water speed and the ocean current speed. The AUV still water speed component model is constructed as follows: According to the above equation, we can find the actual speed of AUV in grid A on the three coordinate axes: According to the above equation, we can find the actual speed of AUV in grid B on the three coordinate axes: Finally, the actual speed of AUV in grid A is Finally, the actual speed of AUV in grid B is The time delay of the AUV from the center point a of grid A to the center point b of grid B is calculated as: In the above formula: represents the X-axis still water velocity component of the AUV in grid A; represents the Y-axis still water velocity component of the AUV in grid A; represents the Z-axis hydrostatic velocity component of the AUV in grid A; Represents the ocean current component in the X-axis direction in the A grid; Represents the ocean current component in the Y-axis direction in the A grid; Represents the ocean current component in the Z-axis direction in the A grid; represents the X-axis still water velocity component of the AUV in grid B; represents the Y-axis still water velocity component of the AUV in grid B; represents the Z-axis hydrostatic velocity component of the AUV in grid B; represents the ocean current component in the X-axis direction in the B grid; Represents the ocean current component in the Y-axis direction in the B grid; Represents the ocean current component in the Z-axis direction in the B grid; V represents the still water speed of the AUV; No r( ) represents the normalization operation function; o represents the center point of the contact surface between mesh A and mesh B; a represents the center point of grid A; b represents the center point of grid B; dc ao represents the predetermined direction vector of the AUV traveling from point a to point o; t a,b represents the delay from grid A to grid B; dis ao Indicates the distance from point a to point o; dis ob Represents the distance from point o to point b; represents the actual speed of the AUV in grid A; Indicates the actual speed of the AUV in grid B.

5. The underwater multi-AUV task allocation method based on deep reinforcement learning according to claim 4 is characterized in that: Let grid α u and grid β u For AUV in a pair of sensor nodes n i and n j The uth pair of adjacent grids passed by when driving; Represents the grid α u To Grid β u The delay of n i The first sensor node number in the pair of sensor nodes; n j Number the second sensor node in the pair of sensor nodes; Define the AUV as a pair of sensor nodes n i and n j The travel delay between sensor nodes n i and n j The edge delay weight between sensor nodes n i and n j The calculation formula of the edge delay weight between is as follows: Where: delay i,j represents sensor node n i and n j The edge delay weight between them.

6. The underwater multi-AUV task allocation method based on deep reinforcement learning according to claim 1 is characterized in that: The method comprises the following steps: Step 1: Based on A * The algorithm is used to optimize the AUV's driving trajectory between two different grids where two sensor nodes are located; let open set Indicates the grid where the AUV is currently located The set of all neighbor grids G nei Exclude the grid set formed after the grid has been passed; close set Represents the set of grids that the AUV has passed through before; A * The algorithm will choose open set The grid with the smallest delay cost is used as the next grid to be passed, and this grid is used as the grid where the new AUV is currently located; the above process is repeated until the AUV reaches the grid where the target sensor node is located; Once the target sensor node is reached, close set All grids that the AUV passes through during its trajectory will be saved; then the edge delay between two sensor nodes is calculated based on the trajectory; using A * The algorithm calculates the edge delays between all sensor nodes and finally obtains the edge delay matrix between all sensor nodes; Step 2: Assume that each AUV collects a set of sensor nodes; Assume that a set of sensor nodes is assigned to the AUV, and use the LKH algorithm to find the access path; Abstract the assigned sensor nodes as vertices in the graph, and abstract the edge delay of the AUV navigation trajectory between two sensor nodes as the weight of the edge between the two vertices; The LKH algorithm takes this graph as input; based on this input, it generates an initial feasible path, and let tour be the initial feasible path; The LKH algorithm is used to find the shortest travel path for each AUV, and finally the path delay corresponding to each AUV is obtained; Step 3: Based on the edge delay matrix between all sensor nodes and the path delay corresponding to each AUV; construct an optimized allocation strategy based on the GA algorithm and the N-step DQN algorithm; use the GA algorithm to generate the initial allocation strategy, and the N-step DQN algorithm will perform secondary redistribution based on the initial allocation strategy; the N-step DQN algorithm will start from the first sensor node and allocate sensor nodes according to the optimized allocation strategy, and the sensor nodes that have not been allocated will be allocated according to the initial feasible strategy generated by the GA algorithm; therefore, the estimated path delay of the task allocation can be calculated, and the reward of the environmental feedback is no longer 0. When selecting actions, the ε greedy strategy is executed, and the sensor node is allocated to a certain AUV based on the current optimal task allocation strategy φ with a probability of 1-ε, and randomly allocated to a certain AUV with a probability of ε; Then, it is determined whether all sensor nodes have been allocated. If so, the termination state is reached, otherwise, the termination state is not reached. Different methods are used to calculate the target value according to the termination state and the non-termination state. The optimization allocation strategy algorithm continuously learns and eventually obtains the optimal task allocation strategy.

7. The underwater multi-AUV task allocation method based on deep reinforcement learning according to claim 6 is characterized in that: Step 1 includes the following method steps: Assume that the grid where the AUV is currently located is The grid to be selected is F s Representation Grid Delay cost; AUV needs to The adjacent grid set G nei Select a grid to form a set of grids to be selected open set ; Evaluate open set Each grid The delay cost F s , and select open set The one with the smallest delay cost is selected as the next grid, and Update it to it; at the same time, the previous Add to close set Medium; F s The AUV moves from the grid where the source sensor node is located to the grid The actual delay cost g s and from the grid The heuristic delay estimate h to the grid where the target sensor node is located s Composition; actual delay g s The calculation is expressed by the following formula: g s =g c +t c,s ; Where: t c,s Represents the grid To Grid The time delay between g c Indicates that the AUV moves from the grid where the source sensor node is located to the grid The actual delay of According to AUV The trajectory to the grid where the target sensor node is located is calculated in turn along the three coordinate axes to calculate the heuristic delay evaluation h s ; This trajectory is divided into three sections; the first section is from To the grid with the same X-axis coordinate as the target sensor node The second paragraph is from Along the Y-axis direction, go to the grid with the same Y-axis coordinate as the grid where the target sensor node is located. The last paragraph is from Along the Z axis to the grid where the target sensor node is located; assuming that the grid sets that the AUV passes through in these three segments are and Where Ω x is the difference between the coordinate values ​​of the selected grid and the target grid on the X-axis; Ω y is the difference between the coordinate values ​​of the selected grid and the target grid on the Y axis; Ω z is the difference between the coordinate values ​​of the selected grid and the target grid on the Z axis; The process is described as follows: Where: u represents the set G x The grid number in ; v indicates the set G y The grid number in ; w represents the set G z The grid number in ; t u,u+1 Represents AUV from the set G x The time delay from the uth grid to the u+1th grid; t v,v+1 Represents AUV from the set G y The time delay from the vth grid to the v+1th grid; t w,w+1 Represents AUV from the set G z The time delay from the wth grid to the w+1th grid; The final delay is: F s =g s +h s 。.

8. The underwater multi-AUV task allocation method based on deep reinforcement learning according to claim 6 is characterized in that: Step 2 includes the following method steps: Assume n1, n2, n3 and n4 are vertices in tour; randomly select the initial vertex n1; then, select an edge x1=(n1,n2) in tour as the edge to be removed, and the corresponding edge y1=(n2,n3) not in tour as the connecting edge, and select x2=(n3,n4) in tour as the second edge to be removed, corresponding to y2=(n4,n1) as the second connecting edge; randomly select two edges in the iteration to remove them and replace them with two new edges; generate a new path tour′; if the path delay of tour′ is less than the path delay of tour, it is considered that the bilateral exchange is completed; otherwise, try a three-edge exchange; repeat the above steps until the path delay of tour′ is less than the path delay of tour or the number of exchanged edges is equal to 5; if the algorithm fails to find a better solution when reaching the five-edge exchange, it will select a new initial vertex in the loop and repeat the above steps; continue until every vertex in tour is used as the initial vertex, and finally output the optimal path.

9. The underwater multi-AUV task allocation method based on deep reinforcement learning according to claim 6 is characterized in that: Step 3 includes the following method steps: Formulate the following ε-greedy strategy: Where: j is the sensor node number; π(e|s) represents the ε-greedy action selection strategy; |M| represents the number of selectable actions, and its value is equal to the number of AUVs; e * is the action selected according to the optimal allocation strategy; Represents the estimated Q value calculated based on the current Q network; s j Indicates the state where the first j sensor nodes have been assigned; A represents the state s j The set of optional actions in , i.e., all AUVs to which the j+1th sensor node can be assigned; e represents the current state s j The action selected; i.e., the AUV to which the j+1th sensor node is assigned; Indicates j The corresponding state vector; Indicates j+1 The corresponding state vector; θ represents the parameters of the neural network; In order to make the algorithm converge, ε decreases and tends to 0 as the algorithm iterates; Execute action e to assign the corresponding sensor node to the selected AUV and obtain the corresponding state vector φ(s j+1 ) after j will move to the new state s j+1 ;s j+1 Indicates the state where the first j+1 sensor nodes have been assigned; φ(s j+1 ) means s j+1 The corresponding state vector; Then, determine whether all sensor nodes have been assigned. If so, the terminal state is reached, otherwise the terminal state is not reached. Different methods are used to calculate the target Q value y according to the terminal state and non-terminal state. j ; The specific calculation formula is as follows: Where: Among them, done is used to indicate s j+1 Whether it is terminated, that is, whether all sensors have been allocated; r j Indicates the state is s j The reward of environmental feedback is determined by the path delay of the first j sensor nodes assigned; Indicates that in state s j The target Q value at ; r j (n) Indicates that in state s j The next n sensor nodes are allocated according to the strategy φ, and n path delays are obtained. These delays are accumulated as state s j Rewards; γ represents the discount factor; γ (n) represents the nth power of γ; γ (k) represents γ to the power of k; n and k represent the number of steps in N-step DQN; r j+k+1 Indicates state s j+k+1 Rewards; s j+n Indicates state s j The state after the nth step; e′ represents the action of the target Q network; θ′ represents the parameters of the target Q network; Indicates that the target Q network state is s j+n , Q value when the action is e′; Indicates that the target Q network is in state s j+n When , choose the action with the largest Q value; Indicates state s j+n The corresponding state vector; Assume that the n sensor nodes are assigned only for pre-evaluation rewards, but only the j+1th sensor node is actually assigned and the state is only updated to s j+1 ;at last, It is stored as a single sample in the experience replay buffer D. When there are enough samples in D, samples are extracted from D for loss calculation. The loss function is written as: Where: L i (θ i ) represents the loss of network parameters at the i-th iteration; It means to extract multiple samples from D and find the average value; r represents the portion of reward elements in a single sample in D; U(D) means uniform sampling; θ i represents the network parameter value at the i-th iteration, where i represents the number of iterations; θ′ i represents the network parameters of the target Q network at the i-th iteration; Indicates that the target Q network is in state s j+n The Q value obtained by taking action e′ when Indicates that the target Q network is in state s j+n When , choose the action with the largest Q value; Indicates that the current Q network is in state s j+n The Q value obtained by taking action e when Target network parameter θ′ i Only Q network parameters θ are used every C steps i is updated and is fixed between updates; it updates all parameters of the current Q network θ by backpropagating the gradient of the neural network as follows: Where: Denotes the loss function L i For the parameter θ i The gradient of Indicates that the current Q network is in state s j+n The Q value obtained by taking action e when Denotes the action value function Q with respect to parameter θ i The gradient of The optimization allocation strategy algorithm continuously learns and eventually obtains the optimal task allocation strategy.

10. A device for underwater multi-AUV task allocation method based on deep reinforcement learning, comprising a memory and a processor, characterized in that: The memory is used to store a computer program; the processor is used to execute the computer program and implement the steps of the underwater multi-AUV task allocation method based on deep reinforcement learning as described in any one of claims 1 to 9 when executing the computer program.

Images