Task allocation method for multiple AUV (Autonomous Underwater Vehicle) systems in communication limited scene

Abstract

The invention provides a task allocation method for multiple AUV (Autonomous Underwater Vehicle) systems in a communication limited scene, which belongs to the technical field of AUVs and comprises the following steps: S1, establishing a Markov decision model; s2, designing a strategy estimation module by using a linear regression algorithm; s3, designing a reward function for guiding cooperation according to the strategy estimation module in the step S2; and S4, improving the DQN algorithm of the distributed multi-AUV system. According to the invention, the strategy estimation module is used for estimating the real-time strategy of each AUV, so that the information exchange between the AUVs in a non-communication state is realized, the task target is efficiently and cooperatively accessed by a multi-AUV system, and the overall efficiency and performance are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of AUVs, relates to a method for collecting information of multiple AUVs, and specifically to a method for allocating tasks of multiple AUV systems in a communication-restricted scenario. Background Art

[0002] In recent years, people have become more and more curious about the unknown underwater world, and their interest in underwater territories has also attracted more and more attention. Therefore, it has become increasingly important to explore and understand the marine environment. As an unmanned underwater vehicle, AUV is widely used in military and civilian fields, and plays an irreplaceable role in tasks such as seabed exploration, data collection, life-saving and rescue, and target search. With the increasing demand for underwater tasks, multi-AUV systems have unique advantages over single AUVs in terms of large number of tasks, complex tasks, and collaboration, and have high scalability and stability. However, multi-AUV systems will inevitably face task allocation problems when completing tasks. Only by understanding the number and content of tasks, as well as the number and performance of AUVs, can we formulate allocation strategies, improve the efficiency of task execution, and give full play to the advantages of multi-AUV systems.

[0003] At present, scholars at home and abroad have achieved some results in using AUV for information collection. Qtte et al. solved the task allocation problem in underwater environment by considering various evaluation functions of bidders and auctioneers, including profit maximization, cost minimization, etc. Gyeongtaek et al. proposed a new market-based decentralized algorithm for task allocation in a dynamic environment within a limited communication range, considering the interaction between tasks. In recent years, the research on intelligent algorithms in task allocation has attracted more and more attention from researchers. Among them, bio-inspired intelligent optimization algorithms mainly include particle swarm optimization algorithms, genetic algorithms, and ant colony algorithms. With the extensive research of neural networks, self-organizing maps (SOM) have been successfully applied to underwater task allocation. However, in harsh underwater scenarios, communication between AUVs is severely restricted, communication delays and communication quality seriously affect the cooperation between AUVs, information exchange is very inconvenient, and higher requirements are also placed on the autonomous decision-making ability of a single AUV. Summary of the invention

[0004] In order to solve the problem that the communication between AUVs in harsh underwater scenarios is severely restricted, resulting in inconvenience in information exchange between multiple AUVs, and at the same time to solve the problem that the autonomous decision-making ability of a single AUV is low, the present invention provides a task allocation method for a multi-AUV system in a communication-restricted scenario.

[0005] The technical solution of the task allocation method of a multi-AUV system in a communication-limited scenario of the present invention is:

[0006] A method for task allocation of multiple AUV systems in a communication-limited scenario comprises the following steps:

[0007] S1. Establish a Markov decision model;

[0008] S2. Design a strategy estimation module using a linear regression algorithm. Given a set of input samples and the target value corresponding to each sample, it is necessary to learn the functional relationship between the target value and the input value under a certain loss criterion. When there are new samples, the corresponding target value can be predicted. Given n attributes x = (x1, x2, ..., x n ), x i Where is the value of x on the i-th attribute, and the linear model learns a function to make predictions through the linear combination of attributes;

[0009] S3, designing a reward function to guide collaboration according to the strategy estimation module of step S2;

[0010] S4. Combined with the improved DQN algorithm, through a large amount of training to update the DQN network parameters, in the process of continuously minimizing the loss function, S makes the algorithm converge, and then the algorithm is tested; after obtaining a stably converged algorithm, the obtained stable convergence algorithm is applied to the scenario of underwater information collection with limited communication to complete the underwater multi-target allocation task.

[0011] A further improvement of the technical solution of the present invention is that: the step S1 includes the following steps:

[0012] S1.1, establish AUV set and target sensor set;

[0013] S1.2. Apply the AUV set and target sensor set in S1.1 to the Markov decision process to establish a Markov decision model.

[0014] A further improvement of the above technical solution of the present invention is that: the step S1.1 comprises:

[0015] The set of AUVs is represented by I = {1, 2, ..., i, ..., I}. AUVi is equipped with a positioning system and can observe the positions of other AUVs. At time t, the position information of other AUVs observed by AUVi is represented by the set O t i，AUV = {O t i，AUV1 , …, O t i，AUVj≠I , …, O t i，AUVI} indicates that each AUV consumes energy E per unit time, and the energy E can be provided by the AUV's own engine. Therefore, the effective operation time of the AUV is limited, which is represented by T = E / e;

[0016] The target sensor set is represented by M = {1, 2, ..., m, ..., M}, and the position information of the target sensor observed by AUVi at time t is represented by the set O t i，TAR = {O t i,TAR1 ,…,O t i,TARm ,…,O t i,TARM}express;

[0017] The effective communication radius between the AUV and the target sensor is represented by r, and the distance between the AUVi and the target sensor m is L i，m ; if L i，m ≤R, AUVi will successfully complete the visit to the target sensor m; the successful visit to the task target m is marked as S m =1, otherwise it is marked as S m = 0; when all target sensors are successfully accessed, the task of collecting target information is completed. We define the completion time of collecting target information as T C .

[0018] A further improvement of the above technical solution of the present invention is that: the step S1.2 includes:

[0019] The task allocation process of the multi-AUV system can be regarded as a Markov process, whose tuple is given by (S, A, P, R). In the case of a given system model, the state space of the AUV is S, the action space is A, the transfer function is P, and the reward is R;

[0020] In the Markov process, the state at time t is S t =[O t i，TAR , O t i，AUV ]; Based on the given S t , AUVi selects a target as an action a i，t , strategy π t ={a 1，t , ..., a i，t , ..., a I，t} is the action set of each AUV, which represents the joint strategy of the Multi-AUV system. The reward of the current action is expressed as r t+1 =R(s t , a t ,s t+1 ) indicates that the transfer function P(s t+1 |s t , a t ) determines the AUV to (st , a t ) s under the condition t+1 The transition probability of

[0021] Based on the above analysis, the mission planning of the multi-AUV system can be expressed as the following optimization problem:

[0022]

[0023] The first constraint means that each AUV completes the data collection task within its effective working time, and the second constraint means that all targets should be visited successfully.

[0024] A further improvement of the technical solution of the present invention is that: the step S2 comprises:

[0025] The linear regression model is a supervised learning model;

[0026] f(x)=β1x1+β2x2+…+β n x n +β0

[0027] In Beta i and β0 are all parameters. After learning the parameters, the model can be determined. Given a data set D = (x1, y1), (x2, y2), …, (x m ,y m ), where x is a D-dimensional data set, xi = (x i1 , x i2 ,…,x id ), y i ∈R;

[0028] Multiple linear regression uses a linear combination of x to predict y. The difference between the true value and the predicted value is expressed as:

[0029]

[0030] For each data point, calculate its residual e i The value of is then squared and summed to quantify the error between the fitted summation line and the actual summation line; put it into the formula:

[0031]

[0032] When the sample has n attributes to describe, multiple linear regression is used, and the least squares method can be used for parameter estimation. At this time, the analytical solution of the least squares parameter estimation can be obtained:

[0033]

[0034] In a multi-AUV system, each AUV can observe and save the position information of other AUVs. A single AUV with a decision estimation module will fit the corresponding linear trajectory based on the historical position information of other AUVs in each decision to estimate the next position. We take the latest 5 position information of each AUV as the given data set, and use the direction vector composed of the estimated position and the latest historical position as the estimated value of the AUV decision.

[0035] A further improvement of the technical solution of the present invention is that the reward function of step S3 includes:

[0036] 1) Distance between the AUV and the target: When the AUV approaches the target point, a positive fixed reward is given, and when the target point is not reached, a reward is given using the Euclidean distance from the target point to the current position of the AUV;

[0037]

[0038] Where C1 and C2 are positive numbers, (x, y, z) are the position coordinates of the AUV, (x i ,y i 、z i ) are the corresponding target coordinates;

[0039] 2) Task completion time loss: Design a small negative reward for each step of the AUV;

[0040] R t =-C3T

[0041] Where T is the current total number of steps, and C3 is the possible number;

[0042] 3) Collaboration of multiple AUVs: In the case of limited communication, AUVs cannot communicate directly. A strategy estimation module is designed to estimate the strategies of other AUVs, and then a reward function is designed to guide the cooperation of multiple AUVs to ensure overall efficiency;

[0043]

[0044] In the formula, C4 and C5 are possible numbers, L i is the distance between the assigned AUV and the corresponding target, N c is the number of strategic conflicts of AUV.

[0045] A further improvement of the technical solution of the present invention is that the improved DQN algorithm in step S4 is a distributed extended DQN algorithm, which is as follows:

[0046] The optimal policy for an MDP with unknown transition probabilities and reward functions can be solved using RL. To evaluate policies with discounted long-term rewards, two value functions are defined in RL. The first is called the state-value function, which measures the expected cumulative discounted reward starting from s and after π, and is defined as:

[0047]

[0048] The other value function is called the action-value function and is defined as:

[0049]

[0050] It measures the expected total discounted reward of taking action a at state s and taking action π later. The Bellman equation for the Q function is:

[0051]

[0052] The optimal strategy can achieve the optimal state value function V * (s)=max π V π (s) and the action-value function Q * (s, a) = max π Q π (s, a), the Bellman optimality equation of the optimal Q function is:

[0053]

[0054] A typical DRL method is deep Q learning, which is a value-based DRL method that learns a neural network to approximate the optimal Q function by minimizing the squared time difference error. The loss function is defined as:

[0055]

[0056] Among them, θ Q represents the weight of the neural network, θ Q ' represents the weight of the target network, θ Q Stochastic gradient descent is used for updating, and experience replay is used in DQN as a sample-efficient technique. Q’ Repeat θ periodically Q , the optimal strategy is:

[0057]

[0058] It is possible to solve the optimal strategy for MDPs with discrete action spaces.

[0059] A straightforward way to apply DRL to a multi-AUV system is to have each AUV learn independently and treat the other AUVs as part of the environment. The Q function of one AUV can be calculated as the state transition p(s t+1 |s t ,a t i ) can be decomposed into:

[0060]

[0061] Where p(a t -i |s t )=Π j≠i p(a t j |s t ) and a t -i Indicates except A i The joint action of the external AUV, during the learning process, the change of AUV strategy leads to p(s t+1 |s t ,a t i ) is non-stationary. When using iterative learning of the Q function, the non-stationary component will lead to invalid results.

[0062] Define an extended Q function that includes the actions of other AUVs so that the action distribution can be removed. Specifically, first define an extended Q function Q πi i (s,a -i , a), when the state is s, it measures A i The expected total reward of other AUVs is a -i , A i Take action a i , followed by π i , then, derive A i The Bellman optimality equation for the optimal extended Q function of the strategy is as follows:

[0063]

[0064] Define the auxiliary variable as:

[0065]

[0066] Among them, s t -i Indicates that iThe global state part related to the AUV other than a is a function that needs to be learned. During the learning process, the learning delay is used to accept the next state one step in advance. In addition, due to the uncertainty of state transition, the auxiliary variables can be estimated, and their expectation is:

[0067]

[0068] On this basis, the approximate Bellman optimality equation of the optimal extended Q function corresponding to the optimal strategy of the AUV is derived. Therefore, we first define a G function as:

[0069]

[0070] Therefore, the approximate Bellman optimality equation for the G function is:

[0071]

[0072] By minimizing the TD error with angles, G can be learned i To approximate the G function, the loss function is:

[0073]

[0074] Similarly, an approximate Bellman expectation equation can be derived as:

[0075]

[0076] In order to use the neural network G i Approximate G π i , and its loss function is:

[0077]

[0078] Based on the above optimal G-function learning method, an improved distributed multi-AUVDQN algorithm was designed.

[0079] Due to the adoption of the above technical solution, the technical advances achieved by the present invention include:

[0080] 1. The present invention uses a strategy estimation module to estimate the real-time strategy of each AUV, thereby realizing information exchange between AUVs without communication, thereby completing the efficient and collaborative access of multiple AUV systems to mission targets, greatly improving the overall efficiency and performance.

[0081] 2. Utilizing the strong coupling between the network structure of DQN and the reward function that guides its training, the present invention combines these two ideas to handle the task allocation of moving targets in the same situation without communication. The Q network is extended to the state space of the G network to introduce global information, which not only improves the generalization ability, but also improves the stability of training, and can be better applied to the task allocation of dynamic targets without communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 is a flow chart of a task allocation algorithm for a multi-AUV system based on an estimation module of the present invention;

[0083] Figure 2 This is the task allocation effect diagram of the conventional DQN algorithm;

[0084] Figure 3 This is the task allocation effect diagram of the improved DQN algorithm;

[0085] Figure 4 This is a diagram of the DQN algorithm task allocation effect based on the estimation strategy;

[0086] Figure 5 This is a diagram showing the effect of task allocation of the improved DQN algorithm based on the estimation module of the present invention;

[0087] Figure 6 This is the effect diagram of the improved DQN algorithm task allocation under ideal communication;

[0088] Figure 7 This is a comparison chart of the maximum completion time of different algorithms;

[0089] Figure 8 This is a comparison chart of energy loss of different algorithms. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in combination with specific implementation methods and with reference to the accompanying drawings. In the following description, the description of the public structure and technology is omitted to avoid unnecessary confusion of the concept of the present invention.

[0091] The present invention provides a multi-AUV system task allocation method based on estimation strategy. First, a linear regression algorithm is used to design a strategy estimation module. Secondly, the estimation strategy is used to design a reward function that guides multiple AUVs to cooperate with each other. Finally, combined with the improved DQN algorithm, a large amount of training is performed to update the DQN network parameters. In the process of continuously minimizing the loss function, the algorithm converges, and then the algorithm is tested. After obtaining a stable convergent algorithm, it is applied to the scenario of underwater information collection with limited communication to complete the underwater multi-target allocation task. Specifically including the following contents:

[0092] 1. System Model

[0093] 1. Model of multi-AUV system task allocation problem

[0094] In a three-dimensional underwater scene, multiple AUV systems collaboratively access multiple moving targets within a specified maximum completion time. The multi-AUV system needs to efficiently access all targets and collect their information. In harsh underwater scenes, current underwater communication technologies, including hydroacoustic communication and optical communication, cannot guarantee effective long-distance communication between AUVs. Based on the above analysis, it is assumed that the multi-AUV system cannot communicate directly.

[0095] The set of AUVs is represented by I = {1, 2, ..., i, ..., I}. AUV i is equipped with a positioning system and can roughly observe the positions of other AUVs. The position information of other AUVs observed by AUV i at time t is represented by the set O t i,AUV = {O t i,AUV1 ,…,O t i ,AUVj≠I,…,O t i,AUVI}. It should be noted that due to the limitation of the accuracy of the positioning system, the position information of other AUVs observed is biased. Energy E can be provided by the AUV's own engine, and each AUV consumes energy E per unit time. Therefore, the effective operation time of AUV is limited, which is represented by T = E / e.

[0096] The target sensor set is represented by M = {1, 2, ..., m, ..., M}. Similarly, the position information of the target sensor observed by AUV i at time t is represented by the set O t i,TAR = {O t i,TAR1 ,…,O t i,TARm ,…,O t i,TARM It should be noted that this information is also biased.

[0097] The effective communication radius between the AUV and the target sensor is represented by r, and the distance between AUV i and the target sensor m is L I,m . Assume that if L I,m ≤R, AUV i will successfully complete the visit to the target sensor m. The successful visit of the mission target m is marked as S m =1, otherwise it is marked as S m= 0. When all target sensors are successfully accessed, the task of collecting target information is completed. We define the completion time of collecting target information as T C .

[0098] 2. Markov Decision Process

[0099] The task allocation process of the multi-AUV system can be regarded as a Markov process, whose tuple is given by (S, A, P, R). Given the system model, the state space of the AUV is S, the action space is A, the transfer function is P, and the reward is R.

[0100] In the above Markov process, the state at time t is S t =[O t i,TAR ,O t i,AUV Based on the given S t , AUV i selects a target as an action a i,t Strategy π t ={a 1,t ,……,a i,t ,……,a I,t} is the set of actions for each AUV, which represents the joint strategy of the multi-AUV system. The reward for the current action is r t+1 =R(s t , a t ,s t+1 ) represents the transfer function P(s t+1 |s t ,a t ) determines the AUV to (s t , a t ) s under the condition t+1 The transition probability.

[0101] 3. Question Construction

[0102] Based on the above analysis, the mission planning of the multi-AUV system can be expressed as the following optimization problem:

[0103]

[0104] The first constraint means that each AUV completes the data collection task within its effective working time. The second constraint means that all targets should be visited successfully.

[0105] To accomplish this task, the following issues need to be addressed:

[0106] 1) Due to the lack of reliable centralized information processing units in underwater scenarios, each AUV needs to implement intelligent and autonomous distributed system strategy optimization.

[0107] 2) Communication is limited, AUVs cannot directly exchange information or cooperate with each other.

[0108] 3) Faced with dynamic mission objectives, AUVs need to have the ability to adjust allocation strategies in real time to achieve dynamic cooperation.

[0109] 2. Improved DQN task allocation algorithm for multi-AUV system based on estimation strategy

[0110] 1. Policy estimation algorithm based on linear regression

[0111] The linear regression model is a supervised learning model.

[0112] f(x)=β1x1+β2x2+…+β n x n +β0

[0113] In Beta i and β0 are parameters. After learning the parameters, the model can be determined. Given a data set D = (x1, y1), (x2, y2), ..., (x m ,y m ), where x is a D-dimensional dataset, x i =(x i1 , x i2 ,…,x id ), y i ∈R.

[0114] Multiple linear regression uses a linear combination of x to predict y. When we have a new set of x, we can use this linear equation to predict the approximate y value. However, fitting a straight line or a plane cannot completely and accurately match each point. We can only find the law as much as possible to get a general value. Although errors are inevitable in the fitting process, we hope that the final model can fit all data points to the greatest extent, that is, we hope to minimize the error. The error here is also called residual, that is, the difference between the true value and the predicted value, expressed by the formula:

[0115]

[0116] For each data point, calculate its residual e i The values ​​are then squared and summed to quantify the error between the fitted summed line and the actual summed line. If you put this into a formula:

[0117]

[0118] When the sample has n attributes, multiple linear regression is used, and the least squares method can be used for parameter estimation. At this time, the analytical solution of the least squares parameter estimation can be obtained:

[0119]

[0120] In a multi-AUV system, each AUV can observe and save the position information of other AUVs. A single AUV with a decision estimation module will fit the corresponding linear trajectory based on the historical position information of other AUVs in each decision to estimate the next position. We take the latest 5 position information of each AUV as the given data set, and use the direction vector composed of the estimated position and the latest historical position as the estimated value of the AUV decision.

[0121] 2. Design of reward function to guide multi-AUV system to complete information collection task collaboratively

[0122] 1) Distance between the AUV and the target: When the AUV is close to the target point, a positive fixed reward is given. When the target point is not reached, we use the Euclidean distance from the target point to the current position of the AUV to give a reward.

[0123]

[0124] Where C1 and C2 are positive numbers, (x, y, z) are the position coordinates of the AUV, (x i ,y i 、z i ) are the corresponding target coordinates.

[0125] 2) Mission Completion Time Loss: We require the multi-AUV system to complete the mission as quickly as possible. Therefore, a small negative reward is designed for each step of the AUV to guide it to complete the mission as quickly as possible.

[0126] R t =-C3T

[0127] Where T is the current total number of steps and C3 is the possible number.

[0128] 3) Collaboration of multiple AUVs: In the case of limited communication, where AUVs cannot communicate directly, we design a policy estimation module to estimate the policies of other AUVs, and then design a reward function to guide the cooperation of multiple AUVs to ensure overall efficiency.

[0129]

[0130] In the formula, C4 and C5 are possible numbers, L i is the distance between the assigned AUV and the corresponding target, N c is the number of strategic conflicts of AUV.

[0131] 3. Distributed and Scalable DQN Algorithm

[0132] The optimal policy for an MDP with unknown transition probabilities and reward functions can be solved using RL. To evaluate policies with discounted long-term rewards, two value functions are defined in RL. The first is called the state-value function, which measures the expected cumulative discounted reward starting from s and after π, and is defined as:

[0133]

[0134] The other value function is called the action-value function and is defined as:

[0135]

[0136] It measures the expected total discounted reward from taking action a in the state and taking action π later. The Bellman equation for the Q function is:

[0137]

[0138] The optimal strategy can achieve the optimal state value function V * (s)=max π V π (s) and the action-value function Q * (s, a) = max π Q π (s, a). The Bellman optimality equation for the optimal Q function is:

[0139]

[0140] A typical DRL method is Deep Q Learning (DQN), which is a value-based DRL method. It learns a neural network to approximate the optimal Q function by minimizing the squared temporal difference error (TD error). The loss function is defined as:

[0141]

[0142] Among them, θ Q represents the weight of the neural network, θ Q ' represents the weight of the target network, θ Q Updates are made using stochastic gradient descent. Experience replay is used in DQN as a sample-efficient technique. Q’ Repeat periodically Q . Then, it solves the optimal strategy as:

[0143]

[0144] This method can solve the optimal policy of MDP with discrete action space.

[0145] A straightforward way to apply DRL to a multi-AUV system is to have each AUV learn independently and treat the other AUVs as part of the environment. The Q function of one AUV can be calculated. t+1 |s t ,a t i ) can be decomposed into:

[0146]

[0147] Where p(a t -i |s t )=Π j≠i p(a t j |s t ) and a t -i Indicates except A i During the learning process, the change of AUV strategy may lead to p(s t+1 |s t ,a t i ). Therefore, when using iterative learning of the Q function, the non-stationary component may lead to invalid results.

[0148] To solve this problem, we can define an extended Q function that includes the actions of other AUVs so that the action distribution can be removed. Specifically, we first define an extended Q function Q πi i (s,a -i , a), when the current state is s, it measures A i The expected total reward of other AUVs is a -i , A i Take action a i , followed by π i Then, we can deduce that A i The Bellman optimality equation for the optimal extended Q function of the strategy is as follows:

[0149]

[0150] This problem is solved by using an auxiliary variable to replace the true actions of other AUVs. Generally speaking, the states of other AUVs at two adjacent time steps can partially reveal their behaviors. Therefore, the auxiliary variable is defined as:

[0151]

[0152] where s t -i Indicates that i The global state part related to the AUV other than the state of the AUV is a function that needs to be learned, which implicitly estimates the behavior of others by using the state before and after the behavior of others. In the learning process, we can use the delay of learning to accept the next state one step in advance. In addition, due to the uncertainty of state transition, the auxiliary variable can be estimated, and its expectation is:

[0153]

[0154] On this basis, we can derive the approximate Bellman optimality equation for the optimal extended Q function corresponding to the optimal strategy of the AUV. Therefore, we first define a G function as:

[0155]

[0156] Therefore, the approximate Bellman optimality equation for the G function is:

[0157]

[0158] By minimizing the TD error with angles, G can be learned i To approximate the G function. The loss function is:

[0159]

[0160] Similarly, an approximate Bellman expectation equation can be derived as:

[0161]

[0162] In order to use the neural network G i Approximate G π i , and its loss function is:

[0163]

[0164] Based on the above optimal G function learning method, an improved distributed multi-AUV DQN algorithm is designed. In each time step, each AUV needs to select a target based on its observation of the global state. During the learning process, we use the experience replay method to improve sampling efficiency.

[0165] 3. Performance evaluation of the improved DQN task allocation algorithm for distributed multi-AUV systems based on estimation strategy

[0166] The simulation results of different task allocation algorithms are shown in Table 1.

[0167] Table 1 Simulation results of different task allocation algorithms

[0168]

[0169] Python is used to simulate the use of strategy estimation modules in harsh ocean scenarios to deal with the problem of multi-AUV task allocation when communication is limited. The results are as follows Figure 5 shown.

[0170] In the absence of communication, the DQN algorithm is not competent for the task of assigning multiple AUVs to multiple dynamic targets, which is specifically manifested in the poor ability to complete the task objectives. Comparing the DQN algorithm in the case of communication, we can get the reason that each AUV can only solve the task independently and cannot cooperate with each other due to the lack of communication. Therefore, the overall performance of the multi-AUV system is poor.

[0171] Through correlation analysis, such as Figure 5 As shown in the figure, after adding the strategy estimation module based on the improved DQN algorithm, the multi-AUV system can collaboratively process multi-task goals to ensure the overall maximum benefit. Figure 2 , Figure 3 , Figure 4 , Figure 6 Compared with the situation where a multi-target AUV system faces conflicts in multi-target assignment tasks, it is sufficient to prove the effectiveness of the policy estimation module in the improved DQN network.

[0172] In addition, we generate 100 experiments with random target locations and random AUV initial positions to test the strategy. The maximum task completion time is shown in Figure 7 And the energy consumption curve is as follows Figure 8 As shown in the figure, the maximum task completion time, energy loss, task completion rate, and repeated completion rate of the multi-AUV system are compared. The method is compared with the test results of the communication and non-communication models, and the improved DQN algorithm and the policy estimation algorithm are compared and analyzed. It can be seen that our method has advantages in various evaluation indicators and is basically consistent with the communication results under ideal conditions.

[0173] In the above embodiment, the present invention provides a multi-AUV system task allocation method based on estimation strategy, which uses a strategy estimation module to estimate the real-time strategy of each AUV, thereby realizing information exchange between AUVs in a non-communication state, thereby completing the multi-AUV system to efficiently and collaboratively access the task target, greatly improving the overall efficiency and performance; at the same time, utilizing the strong coupling between the network structure of DQN and the reward function guiding its training, the present invention combines these two ideas to handle the task allocation of mobile targets in the same situation without communication; the Q network is expanded to the state space of the G network to introduce global information, which not only improves the generalization ability, but also improves the stability of training, and can be better applied to the task allocation of dynamic targets without communication.

[0174] The above-described embodiments are merely descriptions of preferred implementations of the present invention, and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made by ordinary persons in the art to the technical solution of the present invention should fall within the protection scope of the present invention, and the technical contents for which protection is sought in the present invention have been fully recorded in the claims.

Claims

1. A task allocation method for a multi-AUV system in a communication-constrained scenario, characterized by: The following steps are included: S1. Establish a Markov decision model; S2. Design a strategy estimation module using a linear regression algorithm. Given a set of input samples and the target value corresponding to each sample, it is necessary to learn the functional relationship between the target value and the input value under a certain loss criterion. When there are new samples, the corresponding target value can be predicted. Given n attributes x = (x1, x2, ..., x n ), x i Where is the value of x on the i-th attribute, and the linear model learns a function to make predictions through the linear combination of attributes; S3, designing a reward function to guide collaboration according to the strategy estimation module of step S2; S4. Combined with the improved DQN algorithm, a large amount of training is performed to update the DQN network parameters. In the process of continuously minimizing the loss function, S makes the algorithm converge, and then the algorithm is tested; After obtaining a stable convergence algorithm, the obtained stable convergence algorithm is applied to the underwater information collection scenario with limited communication to complete the underwater multi-target allocation task.

2. The method for allocating tasks of a multi-AUV system in a communication-limited scenario according to claim 1, characterized in that: The step S1 comprises the following steps: S1.1, establish AUV set and target sensor set; S1.

2. Apply the AUV set and target sensor set in S1.1 to the Markov decision process to establish a Markov decision model.

3. The method for allocating tasks of a multi-AUV system in a communication-limited scenario according to claim 2, characterized in that: The step S1.1 comprises: The set of AUVs is represented by I = {1, 2, ..., i, ..., I}. AUV i is equipped with a positioning system and can observe the positions of other AUVs. At time t, the position information of other AUVs observed by AUV i is represented by the set O t i，AUV = {O t i，AUV1 , …, O t i，AUVj≠I , …, O t i，AUVI } indicates that each AUV consumes energy E per unit time, and the energy E can be provided by the AUV's own engine. Therefore, the effective operation time of the AUV is limited, which is represented by T = E / e; The target sensor set is represented by M = {1, 2, ..., m, ..., M}, and the position information of the target sensor observed by AUVi at time t is represented by the set O t i，TAR = {O t i,TAR1 ,…,O t i,TARm ,…,O t i,TARM }express; The effective communication radius between the AUV and the target sensor is represented by r, and the distance between the AUVi and the target sensor m is L i，m ; if L i，m ≤R, AUVi will successfully complete the access to the target sensor m; The successful visit of task target m is marked as S m =1, otherwise it is marked as S m = 0; when all target sensors are successfully accessed, the task of collecting target information is completed. We define the completion time of collecting target information as T C .

4. The method for allocating tasks of multiple AUV systems in a communication-constrained scenario according to claim 2, characterized in that: The step S1.2 comprises: The task allocation process of the multi-AUV system can be regarded as a Markov process, whose tuple is given by (S, A, P, R). In the case of a given system model, the state space of the AUV is S, the action space is A, the transfer function is P, and the reward is R; In the Markov process, the state at time t is S t =[O t i，TAR , O t i，AUV ]; Based on the given S t , AUVi selects a target as an action a i，t , strategy π t ={a 1，t , ..., a i，t , ..., a I，t } is the action set of each AUV, which represents the joint strategy of the Multi-AUV system. The reward of the current action is expressed as r t+1 =R(s t , a t ,s t+1 ) indicates that the transfer function P(s t+1 |s t , a t ) determines the AUV to (s t , a t ) s under the condition t+1 The transition probability of Based on the above analysis, the mission planning of the multi-AUV system can be expressed as the following optimization problem: The first constraint means that each AUV completes the data collection task within its effective working time, and the second constraint means that all targets should be visited successfully.

5. The method for task allocation of multiple AUV systems in a communication-limited scenario according to claim 1, characterized in that: The step S2 comprises: The linear regression model is a supervised learning model; f(x)=β1x1+β2x2+…+β n x n +β0 In Beta i and β0 are all parameters. After learning the parameters, the model can be determined. Given a data set D = (x1, y1), (x2, y2), …, (x m ,y m ), where x is a D-dimensional data set, xi = (x i1 , x i2 ,…,x id ), y i ∈R; Multiple linear regression uses a linear combination of x to predict y. The difference between the true value and the predicted value is expressed as: For each data point, calculate its residual e i The value of is then squared and summed to quantify the error between the fitted summation line and the actual summation line; put it into the formula: When the sample has n attributes to describe, multiple linear regression is used, and the least squares method can be used for parameter estimation. At this time, the analytical solution of the least squares parameter estimation can be obtained: In a multi-AUV system, each AUV can observe and save the position information of other AUVs. A single AUV with a decision estimation module will fit the corresponding linear trajectory based on the historical position information of other AUVs in each decision to estimate the next position; We take the latest 5 position information of each AUV as the given dataset, and take the direction vector composed of the estimated position and the latest historical position as the estimated value of the AUV decision.

6. The method for task allocation of multiple AUV systems in a communication-limited scenario according to claim 1, characterized in that: The reward function of step S3 includes: 1) Distance between the AUV and the target: When the AUV approaches the target point, a positive fixed reward is given, and when the target point is not reached, a reward is given using the Euclidean distance from the target point to the current position of the AUV; Where C1 and C2 are positive numbers, (x, y, z) are the position coordinates of the AUV, (x i ,y i 、z i ) are the corresponding target coordinates; 2) Task completion time loss: Design a small negative reward for each step of the AUV; R t =-C3T Where T is the current total number of steps, and C3 is the possible number; 3) Collaboration of multiple AUVs: In the case of limited communication, AUVs cannot communicate directly. A strategy estimation module is designed to estimate the strategies of other AUVs, and then a reward function is designed to guide the cooperation of multiple AUVs to ensure overall efficiency; In the formula, C4 and C5 are possible numbers, L i is the distance between the assigned AUV and the corresponding target, N c is the number of strategic conflicts of AUV.

7. The method for task allocation of multiple AUV systems in a communication-limited scenario according to claim 1, characterized in that: The improved DQN algorithm in step S4 is a distributed extended DQN algorithm, which is specifically as follows: The optimal policy for an MDP with unknown transition probabilities and reward functions can be solved using RL. To evaluate policies with discounted long-term rewards, two value functions are defined in RL. The first is called the state-value function, which measures the expected cumulative discounted reward starting from s and after π, and is defined as: The other value function is called the action-value function and is defined as: It measures the expected total discounted reward of taking action a at state s and later taking action π. The Bellman equation for the Q function is: The optimal strategy can achieve the optimal state value function V * (s)=max π V π (s) and the action-value function Q * (s, a) = max π Q π (s, a), the Bellman optimality equation of the optimal Q function is: A typical DRL method is deep Q learning, which is a value-based DRL method that learns a neural network to approximate the optimal Q function by minimizing the squared time difference error. The loss function is defined as: in, θ Q represents the weight of the neural network, θ Q' represents the weight of the target network, θ Q Stochastic gradient descent is used for updating, and experience replay is used in DQN as a sample-efficient technique. Q’ Repeat periodically Q , the optimal strategy is: It is possible to solve the optimal strategy for MDPs with discrete action spaces, A straightforward way to apply DRL to a multi-AUV system is to have each AUV learn independently and treat the other AUVs as part of the environment. The Q function of one AUV can be calculated as the state transition p(s t+1 |s t ,a t i ) can be decomposed into: Where p(a t -i |s t )=Π j≠i p(a t j |s t ) and a t -i Indicates except A i The joint action of the external AUV, during the learning process, the change of AUV strategy leads to p(s t+1 |s t ,a t i ) is non-stationary. When using iterative learning of the Q function, the non-stationary component will lead to invalid results. Define an extended Q function that includes the actions of other AUVs so that the action distribution can be removed. Specifically, first define an extended Q function When the state is s, it measures A i The expected total reward of other AUVs is a -i , A i Take action a i , followed by π i , then, derive A i The Bellman optimality equation for the optimal extended Q function of the strategy is as follows: Define the auxiliary variable as: Among them, s t -i Indicates that i The global state part related to the AUV other than a is a function that needs to be learned. During the learning process, the learning delay is used to accept the next state one step in advance. In addition, due to the uncertainty of state transition, the auxiliary variables can be estimated, and their expectation is: On this basis, the approximate Bellman optimality equation of the optimal extended Q function corresponding to the optimal strategy of the AUV is derived. Therefore, we first define a G function as: Therefore, the approximate Bellman optimality equation for the G function is: By minimizing the TD error with angles, G can be learned i To approximate the G function, the loss function is: Similarly, an approximate Bellman expectation equation can be derived as: In order to use the neural network G i Approximate G π i , and its loss function is: Based on the above optimal G-function learning method, an improved distributed multi-AUVDQN algorithm was designed.

Images