Multi-AUV formation and obstacle avoidance method for generative adversarial interaction self-imitation learning

Abstract

The invention discloses a multi-AUV formation and obstacle avoidance method for generative adversarial interactive self-imitation learning, and the method is based on a plurality of AUVs, each AUV is provided with a strategy network, a value network and a discriminator, each AUV is also provided with a temporary track pool, each AUV is provided with an expert demonstration track, and the expert demonstration track is provided with an expert demonstration track. The multi-AUV formation and obstacle avoidance method comprises the following steps: performing formation training on each AUV: performing formation on all AUVs, and training a strategy model of each AUV according to a formation sequence; the strategy model training method of the AUV comprises the steps that the AUV is placed in a target environment to move, and local observation of a time step # imgabs0 # is obtained; a strategy trajectory # imgabs1 # is generated; and extracting a certain number of state action pairs from the strategy trajectory # imgabs2 # and the expert demonstration trajectory # imgabs3 # to train a discriminator # imgabs4 # of the AUV. According to the multi-AUV formation and obstacle avoidance method for generative adversarial interaction self-imitation learning, in the multi-AUV strategy model training process, each AUV makes a decision only through local observation of the AUV, and the problems of communication delay, data loss and the like caused by frequent information exchange are greatly reduced.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a multi-AUV (autonomous underwater vehicle) formation and obstacle avoidance method for generating adversarial interactive self-imitation learning. Background Art

[0002] Most of the earth's surface is occupied by the ocean, and human cognition and utilization of marine resources are currently very limited. With the increasing scarcity of exploitable resources on land, the development and utilization of marine biological resources and mineral resources has gradually become a strategic goal of all countries. AUV (autonomous underwater vehicle) has a large range of activities, does not require a large amount of surface support, and does not require human intervention. It is a very suitable tool for seabed detection, identification, search and other tasks. Due to the limited detection range and energy storage of a single AUV, coupled with the increasing complexity of underwater tasks, especially when performing larger-scale tasks, multiple AUVs are required to work together to complete tasks that a single AUV cannot complete. Formation control and obstacle avoidance are core technologies that ensure that multiple AUVs can successfully complete complex underwater tasks.

[0003] The continuous development of Multi-Agent Reinforcement Learning (MARL) provides a new solution for improving the control of multiple AUVs in uncertain ocean environments. In multi-agent reinforcement learning tasks, each agent interacts with the environment, makes decisions based on the current state, and obtains environmental rewards to update the agent's strategy, thereby continuously improving the accuracy of the agent's decision-making.

[0004] Imitation learning can be combined with other learning mechanisms to improve its learning efficiency. According to the different combined learning mechanisms, imitation learning can be divided into three categories: behavioral cloning, inverse reinforcement learning, and generative adversarial imitation learning. Behavioral cloning (BC) establishes a direct mapping between the action space and state space of the agent through expert demonstration, similar to the mapping between feature values ​​and labels in supervised learning. The advantage of this method is that the framework structure is simple and easy to implement, but its training results require a large amount of expert demonstration data to support it, and the accuracy of the expert demonstration is high. In addition, the generalization ability is weak, and it is difficult to generalize the strategy to an unknown environment.

[0005] Inverse reinforcement learning (IRL) uses expert demonstration as the optimal solution in the task to infer the reward function, and then uses the obtained reward function to train the agent through reinforcement learning methods, so that the performance of the strategy in the task is close to the expert demonstration effect. This method can obtain a specific reward function, so that the trained strategy can also achieve good performance in other environments. However, in most inverse reinforcement learning methods, it is usually necessary to establish a model to solve a series of planning or reinforcement learning problems in the inner loop, which seriously limits the performance of inverse reinforcement learning methods for robot control in large and complex tasks. Summary of the invention

[0006] Aiming at the technical problems in the prior art of difficulty in collecting excellent demonstration trajectories and poor strategy stability in the process of multi-agent learning, the present invention proposes a multi-AUV formation and obstacle avoidance method for generating adversarial interactive self-imitation learning, which can solve the above problems.

[0007] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0008] A multi-AUV formation and obstacle avoidance method for generating adversarial interactive self-imitation learning, the multi-AUV formation and obstacle avoidance method is based on multiple AUVs, each AUV is respectively deployed with a policy network, a value network and a discriminator, each AUV also has a temporary trajectory pool, each AUV has an expert demonstration trajectory, and the multi-AUV formation and obstacle avoidance method includes:

[0009] Formation training of each AUV includes: forming all AUVs into a formation and training the strategy model of each AUV in the formation order;

[0010] The AUV strategy model training method includes:

[0011] Place the AUV in the target environment and move it to obtain local observations o at time step t i，t , where f represents the number of the AUV;

[0012] The local observation o i，t Input to the AUV’s policy network Get action a i，t ;

[0013] Execute action a i，t , obtain the local observation of the next time step t+1 until all actions of the current round are completed;

[0014] Generate a collection of state-action pairs for the current round, which is the strategy trajectory τ i :

[0015] τ i =[(o i，1 , a i，1 ), (o i，2 , a i，2 ),…,(o i,t , a i，t )];

[0016] From the policy trajectory τ i and expert demonstration trajectory τ iE A certain number of state-action pairs are extracted from the training set to train the discriminator of the AUV.

[0017] The round discriminator All rewards output update the AUV’s policy network and value network

[0018] The multi-AUV formation and obstacle avoidance method also includes the strategy trajectory τ i Demonstrate trajectory with experts iE For comparison, if the strategy trajectory τ i Better than expert demonstration trajectory τ iE , then τ i Stored in temporary trajectory pool τ iT Otherwise, proceed to the next round;

[0019] When the temporary trajectory pool τ iT When the storage limit is reached, the current expert demonstration trajectory τ iE Replaced by temporary trajectory pool τ iT , and clear the temporary trajectory pool τ iT .

[0020] Compared with the prior art, the advantages and positive effects of the present invention are:

[0021] The multi-AUV formation and obstacle avoidance method of the present invention generates adversarial interactive self-imitation learning. During the multi-AUV strategy model training process, each AUV is respectively deployed with a strategy network, a value network and a discriminator. Each AUV makes decisions only through its own local observations, which greatly reduces the communication delay, data loss and other problems caused by the need to frequently exchange information.

[0022] In order to make the AUV training effect break through the limitation of the expert demonstration level, each AUV also has a temporary trajectory pool, which is used to store the excellent trajectories generated by the strategy during the training process. The evaluation criteria for whether the strategy-generated trajectory meets the storage criteria for the temporary trajectory pool are given by human experts. When the temporary trajectory pool reaches the storage limit, it means that a certain number of excellent trajectories have been collected, and these trajectories are better than the expert demonstrations at the current stage. At this time, the temporary trajectory pool is used to replace the current expert demonstration temporary trajectory pool, and the excellent trajectory collection for the next stage is cleared and entered. The AUV training effect is improved by the continuous replacement of expert demonstrations.

[0023] After reading the specific embodiments of the present invention in conjunction with the accompanying drawings, other features and advantages of the present invention will become more clear. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A flowchart of an embodiment of a method for generating a multi-AUV formation and obstacle avoidance system for counter-interactive self-imitation learning in the present invention.

[0026] Figure 2 It is a task diagram of an embodiment of the present method.

[0027] Figure 3 is a schematic diagram of the detection range of a sonar sensor in one embodiment of the method of the present invention;

[0028] Figure 4 5 schematic diagrams of deploying a leader AUV and a follower AUV in one embodiment of the method of the present invention;

[0029] Figure 5a It is a schematic diagram of a first obstacle avoidance task route provided in an embodiment of the method of the present invention;

[0030] Figure 5b is a schematic diagram of a second obstacle avoidance route task provided in an embodiment of the method of the present invention;

[0031] Figure 5c is a schematic diagram of a third obstacle avoidance route task provided in an embodiment of the method of the present invention;

[0032] Figure 6 is a route comparison diagram of a leader AUV and a follower AUV in one embodiment of the method of the present invention;

[0033] Figure 7 is a route comparison diagram of a leader AUV and a follower AUV in another embodiment of the method of the present invention;

[0034] Figure 8 is a learning curve and confidence interval diagram of the pilot AUV in each experiment in an embodiment of the method of the present invention;

[0035] Fig. 9 is a learning curve and a confidence interval diagram of the leader AUV and the follower AUV in each experiment in an embodiment of the method of the present invention;

[0036] Fig.10 is a schematic diagram of trajectories generated by two AUVs on a first obstacle avoidance mission route in an embodiment of the method of the present invention;

[0037] Fig.11 A schematic diagram of trajectories generated by two AUVs on a second obstacle avoidance mission route in an embodiment of the method of the present invention;

[0038] Fig.12 A schematic diagram of trajectories generated by two AUVs on a third obstacle avoidance mission route in an embodiment of the method of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Embodiment 1: This embodiment proposes a multi-AUV formation and obstacle avoidance method for generating adversarial interactive self-imitation learning. The multi-AUV formation and obstacle avoidance method is based on multiple AUVs, each AUV is respectively deployed with a policy network, a value network and a discriminator, each AUV also has a temporary trajectory pool, each AUV has an expert demonstration trajectory, and the multi-AUV formation and obstacle avoidance method includes:

[0041] The formation training of each AUV includes: forming all AUVs into a formation, and training the strategy model of each AUV separately according to the formation order.

[0042] The AUV strategy model training method includes:

[0043] Place the AUV in the target environment and move it to obtain local observations o at time step t it, where i represents the number of the AUV.

[0044] The local observation o i，t Input to the AUV’s policy network Get action a i，t .

[0045] Execute action a i，t , obtain the local observation of the next time step t+1 until all actions of the current round are completed.

[0046] Generate a collection of state-action pairs for the current round, which is the strategy trajectory τ i :

[0047] τ i =[(o i，1 , a i，1 ), (o i，2 , a i，2 ),…,(o i,t , a i,t )].

[0048] From the policy trajectory τ i and expert demonstration trajectory τ iE Extract a certain number of state-action pairs to train the AUV discriminator

[0049] The round discriminator All rewards output update the AUV's policy network and value network

[0050] The multi-AUV formation and obstacle avoidance method also includes the strategy trajectory τ i Demonstrate trajectory τi with experts iE For comparison, if the strategy trajectory τ i Better than expert demonstration trajectory τ iE , then τ i Stored in temporary trajectory pool τ iT Otherwise, proceed to the next round.

[0051] When the temporary trajectory pool τ iT When the storage limit is reached, the current expert demonstration trajectory τ iE Replaced by temporary trajectory pool τ iT , and clear the temporary trajectory pool τ iT .

[0052] In order to make the AUV training effect break through the limitation of the expert demonstration level, each AUV also has a temporary trajectory pool for storing excellent trajectories generated by the strategy during the training process. The evaluation criteria for whether the strategy-generated trajectory meets the requirements for storage in the temporary trajectory pool are given by human experts. When the temporary trajectory pool reaches the storage limit, it means that a certain number of excellent trajectories have been collected, and these trajectories are better than the expert demonstrations at the current stage. At this time, the temporary trajectory pool is used to replace the current expert demonstration temporary trajectory pool, and the excellent trajectory collection at the next stage is cleared and entered. The improvement of AUV training effect is achieved through the continuous replacement of expert demonstrations.

[0053] Each AUV has an independent strategy network π i , and can be obtained through the current environmental state s t Select action a i,t ~π i (a i |s=s t ), and obtain their respective corresponding rewards r i，t The ultimate goal of multi-AUV reinforcement learning is to enable each AUV to maximize its respective discounted cumulative reward:

[0054] According to the characteristics of multi-AUV formation and obstacle avoidance tasks and underwater environment, the method of this embodiment adopts the communication framework of decentralized training + decentralized decision-making (Decentralized Training with Centralized Execution, DTDE). In this communication framework, each intelligent agent is assigned to a policy network and a value network, and is not restricted by the central controller. In this framework, each AUV is assigned to a policy network π (a i |o i θ i ) and a value network v(o i ;ω i ). During the training process, the i-th agent uses the equipped sensors to obtain the local observation o of the current time step t i，t , and then input the local observations into the strategy network and the value network to obtain the action decision a of the current step respectively. i，t Each AUV makes decisions based on its own local observations, which greatly reduces the communication delays and data loss caused by the need to frequently exchange information.

[0055] In some embodiments, the method for training each AUV in formation includes:

[0056] The strategy model of the first AUV is trained. This AUV is used as the pilot AUV (hereinafter referred to as the pilot AUV). The AUV is trained according to the strategy model training method of the AUV until convergence and its strategy model is saved.

[0057] Train the strategy models of other AUVs (hereinafter referred to as follower AUVs) except the first AUV, load the strategy models of all the AUVs in front of it, add the distance and angle between the AUV and the AUV in front and adjacent to it to the local observation of the AUV, train according to the AUV's strategy model training method until convergence and save its strategy model.

[0058] In the method of training each AUV in formation, first load the strategy model AUV1 of the first AUV, that is, the simulation model of the leading AUV into the ocean environment, train it until convergence and save the strategy model. Next, load the strategy model AUV2 of the second AUV, add the simulation model of AUV2 to the already trained strategy model, and continue to train AUV2. The performance of AUV2's strategy π2 will be affected by the state information such as the distance and angle between AUV1 and AUV2. Similarly, on the basis of loading the strategy models of AUV1, AUV2...AUVi-1, add the simulation model of AUVi, and continue to train AUVi. The strategy π of the i-th AUV i The performance of the AUV will be affected by other AUV strategies.

[0059] In some embodiments, when training the strategy models of other AUVs except the first AUV, the strategy network of the AUV is obtained by using Nash equilibrium.

[0060] In the complex training relationship of multiple AUVs, the most common outcome is the Nash Equilibrium (NE), which can be defined as:

[0061]

[0062] In Nash equilibrium, if other AUVs do not change their strategies, the i-th AUV cannot obtain a higher discounted cumulative return no matter how it changes its strategy, then the strategy of this AUV is considered to be π i The optimal strategy has been reached under the current circumstances That is to say, in the process of multi-AUV formation and obstacle avoidance, the learning process of each AUV needs to achieve a dynamic balance.

[0063] like Figure 1 As shown, a flow chart of a multi-AUV formation and obstacle avoidance method for generating adversarial interactive self-imitation learning is provided in an embodiment of the present invention.

[0064] Specifically, the input is the expert demonstration τ of each AUViE ~π iE , randomly initialize the policy network, value network and discriminator policy network parameters θ i , and the value network parameter φ i , initialize the temporary trajectory pool τ iT , with a capacity of N. During the iteration process, each AUV uses its own sensor to obtain the local observation o at the current time step i,t ; Each AUV uses a strategy network According to the local observation o of each current step i，t , select and execute action a i,t ; The state-action pairs (o i,t , a i，t ) is stored in the corresponding strategy trajectory set τ i Middle; Human experts evaluate the strategy trajectory of this round τ i Whether to add to the temporary track pool τ iT ; If the temporary trajectory pool τ iT When the storage limit N is reached, the temporary trajectory pool τ iT Medium data coverage to expert demonstrationτ iE And initialize the temporary trajectory pool τ iT ; Use the ADAM optimizer to update the parameters ω of each discriminator i ; Using each discriminator Get rewards for each AUV in this round based on the strategy trajectory

[0065] During training, the IPP0 algorithm is used to learn a decentralized policy network At the same time, using local observation o i,t And GAE to train a value network Vi. From the strategy trajectory τi and the current expert demonstration τ iE A certain number of state-action pairs are extracted from the training set to train the discriminator of the i-th AUV.

[0066] In some embodiments, the discriminator In the step, the loss function is:

[0067]

[0068] in, Indicates that all state-action pairs (o, a) are real data expectations, Indicates that all data is generated data expectations.

[0069] Discriminator of the i-th AUV It will be based on the state-action pairs in all strategy trajectories of this round.

[0070] In some embodiments, the discriminator The method to output rewards is:

[0071]

[0072] Representation Discriminator According to the state action pair (o i,t , a i,t ) is the reward at time step t.

[0073] All rewards for this round output by the discriminator of the i-th AUV will be used to train the policy network and value network V i .

[0074] In some embodiments, the policy network and value network V i In the update step, the IPPO algorithm is used for update, and the total loss function is expressed as:

[0075]

[0076] in, represents the loss function of updating the policy network, represents the loss function for updating the value network, represents the regularization of the policy network entropy, and c1 and c2 are proportional coefficients.

[0077] For multi-task problems, a method of classifying and extracting expert trajectories is proposed to ensure that the discriminator can give accurate reward values. First, each AUV will get the trajectory τ demonstrated by the expert iE In each round of training, the i-th AUV uses its own sensors to obtain the local observation o at the current time step t. i，t , input local observations into the policy network You can get action a i，t , after executing the selected action, the i-th AUV will obtain the observation of the next time step t+1, and repeat the cycle until the end of time. After t arbitrary time steps, the collection of all state-action pairs generated by the i-th AUV is generated:

[0078] τ i =[(o i，1 , a i，1 ), (o i，2 , a i，2 ),…,(o i,t , a i,t )].

[0079] Demonstration from experts i ~π i and the strategy trajectory τ iE ~π iE Extract a certain number of state-action pairs to train the discriminator of the i-th AUV All rewards for this round output by the discriminator of the i-th AUV will be used to train the policy network and value network

[0080] In some embodiments, the AUV's policy model makes decisions based on a partially observable Markov decision process.

[0081] In this embodiment, a partially observable Markov decision process is used. For reinforcement learning in an environment where the agent can obtain reliable state signals, methods based on Markov decision processes (Markov Decision Process, MDP) have achieved many successes. However, in practical applications, due to the limited detection capabilities of the sensors equipped by the agent, it is impossible to accurately obtain global observation information. In this case, the partially observable Markov decision process (Partially Observable Markov Decision Process, POMDP) ​​allows the agent to make reliable decisions while obtaining incomplete observations. As an extension of POMDP, the decentralized partially observed Markov decision process (Decentralized Partially Observable Markov Decision Processes, Dec-POMDPs) models multiple agents under the same environmental setting conditions, allowing multiple agents to make decisions separately on the premise of obtaining partial environmental observations to obtain the optimal collaborative strategy.

[0082] A decentralized partially observable Markov decision process can be defined as a tuple Where n represents the number of agents in the environment. The global observation is represented by a finite state space array S.

[0083] A=×iA i represents the joint action space, where A i represents the action space of the ith agent. The joint action of all agents at time t can be expressed as a t = 1，t , a 2，t , …, a n,t >, and each agent can only obtain its own action value.

[0084] T: S×A×S→P​T (S)∈[0, 1] represents the state transfer function, which describes the random influence of the joint action on the global state space. Therefore, the transition probability of the next global observation can be obtained based on the global observation and joint action at the current moment, that is:

[0085]

[0086] O=× i O i Then it represents the set of joint local observations, O i represents the local observation space of the ith agent. The joint local observation obtained during the interaction between all agents and the environment at time t can be expressed as o t = <o 1，t , o 2，t ,…,o n,t >, and each agent can only obtain its own local observations.

[0087] f: A×S→Pf(0)∈[0,1] represents the observation function, which describes how the agent extracts local observations from the global observation space. In practical applications, it represents the observations that can be obtained within the sensor detection range, expressed as a probability distribution mapping from the current global observation to the joint local observation, that is:

[0088]

[0089] Represents the initial state distribution of each agent in the environment at time t=0.

[0090] It represents the joint reward function of each agent, which can be represented by the global observation and joint action at the current moment and the global observation at the next moment:

[0091]

[0092] In a distributed observable Markov decision process, the ultimate goal is to maximize the joint strategy π=<π1,π2,…,π n > Any substrategy π i The cumulative expected return from time t to time T, that is:

[0093]

[0094] Among them, γ∈[0,1] is a decay factor, which indicates the correlation between the current cumulative expected return and the reward obtained in the future. If γ is smaller, it means that the agent pays more attention to the reward obtained at the current moment; conversely, it means that the agent also values ​​the performance in the future.

[0095] Furthermore, in multi-agent reinforcement learning, multiple agents interact with the environment at the same time. Each agent i has an independent strategy network π i , and can be obtained through the current environmental state s t Select action a i,t ~π i (a i |s=s t ), and obtain their respective corresponding rewards r i，t The ultimate goal of multi-agent reinforcement learning is to enable each agent to maximize its own discounted cumulative return: However, in a multi-agent system, the strategy π of the i-th agent i The performance of the agent will be affected by the strategies of other agents. Under the complex relationship of multiple agents, the most common solution is the Nash Equilibrium (NE), which can be defined as: In Nash equilibrium, if other agents do not change their strategies, the i-th agent cannot obtain a higher discounted cumulative return no matter how it changes its strategy, then the strategy of the agent is considered to be i The optimal strategy has been reached under the current circumstances That is to say, in multi-agent reinforcement learning, the learning process of each agent needs to achieve a dynamic balance.

[0096] Combining all the above technical solutions, the advantages and positive effects of this embodiment are as follows: This solution combines the reinforcement learning framework of multi-AUV formation and obstacle avoidance, the communication framework of decentralized training and decentralized decision-making, and the generative adversarial self-imitation learning method with human feedback to form a generative adversarial interactive self-imitation learning multi-AUV formation and obstacle avoidance method. The present invention uses the synergy of intelligent agent demonstration and human evaluation, which can refer to expert demonstration and use human knowledge, get rid of the constraints of demonstration quality, and achieve learning effects that surpass expert demonstrations, thereby effectively learning better strategies, improving the stability of strategies, and reducing the cost of collecting expert demonstrations in practical applications.

[0097] The technical effects of the present invention are described in detail below in conjunction with experiments.

[0098] In order to verify the feasibility of the MAGAISIL algorithm in multi-AUV formation and obstacle avoidance control, this experiment deployed a reasonable state space and action space for the multi-AUV system. In addition, in order to visualize the training effect, a simple reward function was designed as an evaluation criterion, and the expert trajectories of different levels were specified through the reward function. We designed three tasks of different difficulty levels. The experimental environment was mainly built through the ROS robot operation platform and Gazebo simulation software. At the same time, we used UUVSimulator to simulate the deployment of multi-AUV systems and sonar sensors.

[0099] (I) State space setting

[0100] Figure 2 This is a schematic diagram of the multi-AUV formation and obstacle avoidance task. The black square in the figure represents an obstacle with a width of 0.5 meters; d O Indicates the distance between obstacles, which is set to 30 meters in the task; (x L ,y L ) is the current position coordinate of the pilot AUV; (x F ,y F ) is the current position coordinate of the follower AUV; the blue sector and green sector represent the sonar sensor detection range of the navigator AUV and the follower AUV respectively. For the follower AUV, in addition to keeping a distance from obstacles, it also needs to keep a certain distance from the navigator AUV, so its observation can be divided into two parts: the observation of tracking the navigator AUV Observation of obstacles detected by sonar

[0101] As for the formation part, for the follower AUV, the observation of its tracking of the leader AUV can be expressed as g F It represents the straight-line distance between the follower AUV and the leader AUV. The calculation formula is as follows:

[0102]

[0103] a F It represents the heading deviation of the follower AUV, that is, its current heading a FH The angle between the current position and the straight line connecting the position of the leader AUV can be expressed as follows:

[0104]

[0105] As for the obstacle avoidance part, this experiment uses sonar sensors to detect the position of obstacles in real time, so that the AUV can maintain a safe distance from the obstacles. The angle range of the sonar is set to The detection distance is set to 33 meters, and the detection beam is set to 600 beams. This experiment uses sonar sensors to detect the location of obstacles in real time, so that the AUV can maintain a safe distance from the obstacles. The angle range of the sonar is set to The detection distance was set to 25 meters, and the detection beam was set to 600 beams. In order to enable each AUV to accurately obtain the position and angle of obstacles, we processed the sonar information accordingly.

[0106] like Figure 3 As shown in the figure, the opening and closing angle of the sonar sensor is 120°, and the total detection area is evenly divided into 6 small sector intervals of 20° each. These sector intervals are [-60°, -40°], [-40°, -20°], [-20°, 0°], [0°, 20°], [20°, 40°] and [40°, 60°]. The shortest distance information to the obstacle detected in each small area is input into the observation space as the observation vector, so that the AUV can accurately determine the location and angle of the current obstacle, so that it can accurately avoid it. For the navigator AUV, the observation of the obstacle detection part of its sonar is its observation space. L , which can be expressed as

[0107] o L ={d L1 , d L2 , d L3 , d L4 , d L5 , d L6}#

[0108] For the follower AUV, the observations of the tracking leader AUV and the sonar obstacle detection part are combined to obtain the observation space o F , which can be expressed as

[0109] o F = {g F , a F , d F1 , d F2 , d F3 , d F4 , d F5 , d F6}#

[0110] All vectors in the observation spaces of the two AUVs are normalized to the range of [-1, 1] to ensure that the input values ​​are within a reasonable range and can be mapped within the non-saturated range of the nonlinear activation function of the neural network to ensure the stability of the training.

[0111] Action Space Settings

[0112] The experimental task in this chapter still requires the AUV to be carried out on a two-dimensional plane, so the rudder blades 2 and 4 that control the AUV to float and sink are still set to 0. Figure 4 As shown in the figure, the action space of the leader AUV and the follower AUV is set the same, and each AUV is deployed with 5 actions: 2 left turns with different angles, 1 straight action, and 2 right turns with different angles. The propeller speed of these five actions is 300 revolutions per minute.

[0113] Task Settings

[0114] In the experiments of this chapter, three different obstacle avoidance tasks were also designed, among which Task 1 was used for algorithm training, and Tasks 2 and 3 were used to test the adaptability of the algorithm. It should be noted that neither the leader AUV nor the follower AUV obtains the position information of the obstacle in advance before performing the task, but adjusts the position to maintain a safe distance from the obstacle based on the information obtained by the sonar sensor during the task.

[0115] Task 1

[0116] The relevant settings for Task 1 are shown in Figure 5(a). The gray lines in the figure represent obstacles with a width of 0.5 meters, which are set from (0,30) to (60,30), (0,0) to (90,0), (60,30) to (60,110), (90,0) to (90,80), (60,110) to (180,110), (90,80) to (150,80), (180,110) to (180,30), (150,80) to (150,0), (180,30) to (240,30) and (150,0) to (240,0). The distance between obstacles is 30 meters. Because the multi-AUV formation and obstacle avoidance task is designed to be implemented on a two-dimensional plane, the obstacles must be set at a suitable height to ensure that both AUVs can accurately detect them. The blue dot at (1,0) and the red dot at (18,0) in the figure represent the mission starting points of the leader AUV and the follower AUV, respectively. In this mission, the leader AUV needs to stay in the middle of the obstacles on both sides to pass through the obstacle area, while the follower AUV needs to track the leader AUV 18 meters behind it while keeping a safe distance from the obstacles on both sides.

[0117] In order to ensure the feasibility of the experiment and avoid too much meaningless exploration by the two AUVs, the mission is considered to have failed if the following situations occur during the experiment:

[0118] The pilot AUV is too close to the obstacle, i.e. |d L |≤2.

[0119] The follower AUV is too close to the obstacle, i.e. |d L |≤2.

[0120] The follower AUV keeps too close or too far away from the leader AUV when tracking it, i.e. |a L |<3 or |a F |>33.

[0121] The heading deviation of the follower AUV is too large, that is,

[0122] Once one of the two AUVs meets the above conditions, the mission of this round will be terminated, and the two AUVs will be reset to the initial position and start the next round of mission.

[0123] Task 2

[0124] In order to test whether the trained model can successfully complete the task in a more complex setting, we set more obstacles on the basis of Task 1. This task is called Task 2, and its relevant settings are shown in Figure 5(b). Four rectangular obstacles with a length of 5 meters and a width of 20 meters are set at (87.5, 40), (62.5, 70), (177.5, 70) and (152.5, 40), and three rectangular obstacles with a length of 20 meters and a width of 5 meters are set at (100, 82.5), (120, 107.5) and (140, 82.5). The purpose of Task 2 is to add obstacles to form additional interference in order to explore the adaptability of the model.

[0125] Task 3

[0126] Task 3 changes the placement angle of the boundary obstacles and increases the total length of the obstacle area, as shown in Figure 5(c). In Task 3, obstacles with a width of 0.5 meters are set from (0,30) to (60,30), (0,0) to (70,0), (60,30) to (110,80), (70,0) to (120,50), (110,80) to (130,80), (120,50) to (170,0), (130,80) to (180,30), (170,0) to (190,0), (180,30) to (230,80), (190,0) to (240,50), (230,80) to (300,80), and (240,50) to (300,50). The purpose of Task 3 is to test whether the model can successfully complete the formation and obstacle avoidance tasks in obstacle areas with different angles and in longer tasks.

[0127] Reward function design

[0128] During the training process of the MAGAISIL algorithm, the rewards output by the discriminators of the two AUVs can only be used as training data and cannot be used to evaluate the performance of the AUVs in the task. Therefore, in this section, a simple reward function is designed for the two AUVs according to the task requirements, but it is not used for AUV training, but only as an evaluation criterion for AUV performance.

[0129] Reward design for the pilot AUV

[0130] In the multi-AUV formation and obstacle avoidance mission, the leader AUV needs to keep its position between the obstacles on both sides to pass through the mission area. It is known that in mission 1, the distance d between the obstacles on both sides is O is 15 meters, and the opening angle of the AUV sonar sensor is After conversion, it can be known that when the shortest distance d between the AUV and the obstacle L Keeping it at about 17.3 meters can ensure that the pilot AUV is in the middle of the obstacle, so its reward function r L It can be defined as:

[0131]

[0132] Reward design for follower AUV

[0133] Unlike the leader AUV, the follower AUV needs to complete the following tasks at the same time in the multi-AUV formation and obstacle avoidance mission: tracking the leader AUV and keeping a safe distance from obstacles on both sides. Therefore, its reward function r F Can be divided into two parts:

[0134]

[0135] in, Represents the reward for the follower AUV tracking the leader AUV. Represents the reward for the follower AUV's obstacle avoidance. Since the follower AUV needs to take care of two tasks at the same time, the design of the reward function cannot be biased towards one side, so the two parts of the reward are multiplied by 0.5 as a balance coefficient. The linear distance g between the follower AUV and the leader AUV F Deviation from course a F To provide rewards, it can be expressed as:

[0136]

[0137] It can be found that when the straight-line distance between the follower AUV and the leader AUV is further away from 18 meters or the heading deviation is greater, this part of the reward will be smaller; conversely, this part of the reward will be larger. is the shortest distance to the obstacle detected by its sonar sensor F After conversion, it can be known that when the shortest distance d between the AUV and the obstacle is F Keeping it at about 17.3 meters can ensure that the follower AUV is in the middle of the obstacle, which can be expressed as:

[0138]

[0139] It can be seen that the greater the deviation of the follower AUV from the middle position of the obstacle, the smaller the reward will be; conversely, the greater the reward will be. Similarly, in the tasks of this chapter, even if the follower AUV triggers the task termination condition, it will still not give a large negative reward value.

[0140] Expert trajectory selection

[0141] Suboptimal Expert Demonstration Acquisition

[0142] In the process of using the IPPO algorithm to train the leader AUV and the follower AUV using environmental rewards, if in a certain round, the cumulative reward obtained by the leader AUV's performance reaches about 335, and the follower AUV can also obtain a cumulative reward of about 355, then the trajectory generated by the two AUVs in this round is considered to be a suboptimal expert demonstration. We selected 4 rounds of such trajectories as suboptimal expert demonstrations, such as Figure 6 shown.

[0143] It can be seen from the trajectory diagram that in these four rounds, although the leader AUV can pass through the obstacle area smoothly, it cannot guarantee that its position remains between the obstacles on both sides. At the same time, although the follower AUV can meet the task conditions of tracking the leader AUV and passing through the obstacle area, the tracking error is significantly larger.

[0144] Get the best expert demonstration

[0145] If in one round, the cumulative reward obtained by the performance of the leader AUV can reach about 400, and the cumulative reward of the follower AUV can also reach about 400, then it is considered that the trajectory generated by the two AUVs in this round reaches the optimal expert demonstration level. Similar to the suboptimal expert demonstration, we also selected 4 rounds of such trajectories as the optimal expert demonstration, such as Figure 7 shown.

[0146] Experimental results analysis

[0147] In order to facilitate the interaction between human experts and AUVs to collect excellent trajectories, the experiments in this chapter train two AUVs separately. For the leader AUV, the following three experiments are conducted:

[0148] Input the suboptimal expert demonstration into the navigator AUV and train it for 1000 rounds in Task 1 using the MAGAISIL algorithm, and save the trained model.

[0149] The best expert demonstration is input into the leader AUV and trained for 1000 rounds in task one using the MAGAIL algorithm, and the trained model is saved.

[0150] The suboptimal expert demonstrations were fed into the leader AUV and trained for 1000 rounds in mission 1 using the MAGAIL algorithm.

[0151] Repeat each of the above experiments three times, and save the cumulative rewards of each round of each experiment to obtain the learning curve and confidence interval of the leader AUV in each experiment, such as Figure 8 shown.

[0152] As can be seen from the figure, the leader AUV trained by the MAGAIL algorithm using the best expert demonstration can achieve the best performance around the 230th round, and the cumulative reward obtained in each round can be stabilized at around 400. The cumulative reward of the leader AUV trained by the MAGAIL algorithm using the suboptimal expert demonstration will fluctuate around the suboptimal expert demonstration level, but cannot reach the level of the best expert demonstration. As for the MAGAISIL algorithm, the reward curve before 150 rounds is very similar to the reward curve generated by the MAGAISIL algorithm based on suboptimal expert demonstrations. This is because at the beginning of training, the expert demonstration pool of the MAGAISIL algorithm has not been replaced, and the suboptimal expert demonstrations are still used for learning; after 150 rounds, the reward curve generated by the MAGAISIL algorithm gradually increases. This is because human experts judge that there are strategy trajectories that are better than the suboptimal expert demonstrations during this period and store them in the temporary trajectory pool. When the trajectory pool reaches the upper limit, the current expert demonstration will be replaced to guide the leader AUV to learn better expert demonstrations; at about 450 rounds, after several rounds of excellent trajectory replacement, the MAGAISIL algorithm can reach the optimal expert demonstration level at this time, and the cumulative reward in each round of training thereafter is maintained at about 400. By comparison, it can be found that under the premise of using suboptimal expert demonstrations, the MAGAISIL algorithm can continuously collect excellent trajectories generated by the strategy, guide the training level of the leader AUV to continuously improve, and thus achieve the effect of optimal expert demonstrations.

[0153] For the follower AUV, the following three experiments are conducted:

[0154] Load the leader AUV model trained by the MAGAISIL algorithm based on the suboptimal demonstration, input the suboptimal expert demonstration into the follower AUV, train it for 1000 rounds in Task 1 through the MAGAISIL algorithm, and save the trained model;

[0155] Load the leader AUV model trained by the MAGAIL algorithm based on the best demonstration, input the best expert demonstration into the follower AUV and use the MAGAIL algorithm to train for 1000 rounds in Task 1, and save the trained model.

[0156] Load the Navigator AUV model trained with the MAGAIL algorithm based on the best demonstration, input the suboptimal expert demonstration into the Navigator AUV and train it for 1000 rounds in Task 1 using the MAGAIL algorithm.

[0157] Repeat each of the above experiments three times, and save the cumulative rewards of each round of each experiment to obtain the learning curve and confidence interval of the follower AUV in each experiment, such as Fig. 9 shown.

[0158] As can be seen from the figure, the reward curve of the follower AUV trained by the MAGAIL algorithm based on the suboptimal demonstration has been rising before 200 rounds, reaching a level close to the suboptimal demonstration and remaining stable around 200 rounds, and cannot surpass the level of the suboptimal demonstration. The reward curve of the follower AUV trained by the MAGAIL algorithm based on the optimal demonstration converges faster, reaching the level of the optimal demonstration and remaining stable in about 100 rounds. Between about 180 rounds and about 380 rounds, after the replacement of expert demonstrations, the training performance of the follower AUV is gradually improving and surpassing the suboptimal expert demonstration; around 400 rounds, the training effect of the MAGAISIL algorithm reaches a level similar to that of the MAGAIL based on the optimal demonstration and remains stable. The comparison of the reward curves shows that the MAGAISIL algorithm can gradually collect excellent trajectories generated by the strategy through the judgment of human experts on the basis of the suboptimal expert demonstration, thereby surpassing the level of the suboptimal expert demonstration and achieving an effect similar to the optimal demonstration.

[0159] Model fitness analysis

[0160] During the training process of Task 1 for the MAGAISIL algorithm based on suboptimal demonstration and the MAGAIL algorithm based on optimal demonstration, the trained strategy model is saved and loaded into Task 1, Task 2, and Task 3 for testing respectively, and the test data of each round is saved and analyzed.

[0161] Model performance in Task 1

[0162] The trained leader AUV and follower AUV models are loaded into Task 1 for testing. During the test, the trajectories generated by the two AUVs are as follows: Fig.10As shown. For the leader AUV, the trajectory generated by the model trained by the MAGAISIL algorithm based on the suboptimal demonstration can achieve a similar effect to the MAGAIL algorithm based on the optimal demonstration. In the task test, it can keep its position between the boundary obstacles on both sides and pass through the obstacle area. For the follower AUV, although the model trained by the MAGAIL algorithm based on the optimal demonstration can smoothly track the leader AUV and pass through the obstacle area, the tracking effect will change significantly when the angle of the boundary obstacle changes, and it cannot balance the tracking and obstacle avoidance tasks well.

[0163] The state information generated by the models trained by the two algorithms in one round in Task 1 was analyzed. The suboptimal expert demonstrations of both the leader AUV and the follower AUV could not complete the tasks with high accuracy in the multi-AUV formation and obstacle avoidance tasks. After training the MAGAISIL algorithm using the suboptimal expert demonstrations, the leader AUV was able to stay about 15 meters away from the obstacle when the obstacle angle did not change, that is, in the middle of the boundary obstacles on both sides; although there would be large fluctuations at the turn, it would make rapid adjustments afterwards to maintain its own safe position.

[0164] After being trained by the MAGAISIL algorithm using the suboptimal expert demonstration, the follower AUV can track the leader AUV at a distance of about 18 meters when there is no need to turn, while ensuring a safe distance of about 15 meters from boundary obstacles and a heading deviation of about 0 rad; after fluctuations occur during turns, it can adjust its position as quickly as possible to return to normal formation and obstacle avoidance tasks.

[0165] In the test of Task 1, the final training effect of the two AUV models trained by the MAGAISIL algorithm based on suboptimal expert demonstrations was able to achieve a level similar to that of the MAGAIL algorithm based on optimal demonstrations in the test.

[0166] Model performance in Task 2

[0167] In order to test the adaptability of the algorithm, the two trained AUV models were loaded into Task 2 for testing. The task trajectories generated during the test are shown in the following figure. Fig.11As shown. It can be seen that the model trained by the MAGAISIL algorithm based on suboptimal demonstrations can ensure that the two AUVs maintain a safe distance between themselves and obstacles when avoiding obstacles and pass through the obstacle area while successfully completing the formation task, even if additional obstacles are added on the basis of Task 1. And the final training effect of the MAGAISIL algorithm based on suboptimal expert demonstrations can reach a similar level to the MAGAIL algorithm based on optimal expert demonstrations. In addition, for the follower AUV, the model trained by the MAGAISIL algorithm can better adjust to the appropriate position after avoiding additional obstacles to complete the tracking of the leader AUV.

[0168] The state information generated by the models trained by the two algorithms in one round in Task 2 was analyzed. In Task 2, the pilot AUV models trained by the two algorithms were able to maintain a distance of about 15 meters between the boundary obstacles on both sides when turning and avoiding obstacles were not required, and they were able to adjust their positions in time after turning to ensure a safe distance until they passed the obstacle area. The MAGAISIL algorithm can use suboptimal expert demonstrations and continuously replace demonstration data with human experts, so that the trained pilot AUV can reach a level similar to that of the MAGAIL algorithm based on optimal demonstrations.

[0169] In Task 2, the addition of additional obstacles will increase the tracking error of the follower AUV, while the model trained by the MAGAISIL algorithm can achieve an effect similar to the optimal expert demonstration through trajectory replacement, which overcomes the problem of low accuracy of the suboptimal expert demonstration to a certain extent. Analysis of the state information shows that the heading deviation and distance information between the model trained by the MAGAISIL algorithm and the obstacle in Task 2 are very close to those of the MAGAIL algorithm based on the optimal expert demonstration; while the distance between the model and the leader AUV is quite different. The reason may be that the total number of rounds limits the number of times the expert demonstration is replaced. Although the replaced demonstration is close to the optimal one, it cannot completely reach the optimal level. In general, the model trained by the MAGAISIL algorithm can adapt well to environments with more obstacles and can complete the task at a higher level.

[0170] Model performance in Task 3

[0171] In order to test whether the formation and obstacle avoidance tasks can be successfully completed in long-distance tasks, the trained model is loaded into the three tasks for testing. At the same time, the placement angle of the boundary obstacles is changed to discuss the model's adaptability to different environments. The task trajectories generated during the test are as follows: Fig.12As shown in the figure. Similar to the results of the first two experiments, the trajectory generated by the leader AUV model trained by the MAGAISIL algorithm based on suboptimal expert demonstrations is very close to that of the MAGAIL algorithm based on optimal expert demonstrations. As can be seen from the figure, the generated trajectories are basically the same. For the follower AUV, the model trained by the MAGAISIL algorithm can better balance the formation and obstacle avoidance tasks at turns, tracking the leader AUV with a smaller deviation while maintaining a safe distance from boundary obstacles.

[0172] The state information generated by the models trained by the two algorithms in one round in Task 3 is analyzed. For the navigator AUV, since the angle of the boundary obstacle is set smaller in Task 3, the distance information between the obstacle and the turn is less volatile. When turning is not required, the navigator AUV model trained by the MAGAISIL algorithm can maintain its position between the obstacles on both sides, and can quickly adjust its position to maintain a safe distance from the boundary obstacles on both sides after turning. As can be seen from the figure, the navigator AUV model trained by the MAGAISIL algorithm based on suboptimal expert demonstrations can reach a very similar level to the MAGAIL algorithm based on optimal expert demonstrations.

[0173] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for a person skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions claimed to be protected by the present invention.

Claims

1. A multi-AUV formation and obstacle avoidance method based on generative adversarial interactive self-imitation learning, characterized in that: The multi-AUV formation and obstacle avoidance method is based on multiple AUVs, each AUV is respectively deployed with a strategy network, a value network and a discriminator, each AUV also has a temporary trajectory pool, each AUV has an expert demonstration trajectory, and the multi-AUV formation and obstacle avoidance method includes: Formation training of each AUV includes: forming all AUVs into a formation and training the strategy model of each AUV in the formation order; The AUV strategy model training method includes: Place the AUV in the target environment and move it to obtain local observations o at time step t i,t , where i represents the number of the AUV; The local observation o i,t Input to the AUV’s policy network Get action a i,t ; Execute action a i,t , obtain the local observation of the next time step t+1 until all actions of the current round are completed; Generate a collection of state-action pairs for the current round, which is the strategy trajectory τ i : τ i =[(the i,1 ,the i,1 ),(the i,2 ,the i,2 ),…,(the i,t ,the i,t )]; From the policy trajectory τ i and expert demonstration trajectory τ iE A certain number of state-action pairs are extracted from the training set to train the discriminator of the AUV. The round discriminator All rewards output update the AUV’s policy network and value network The multi-AUV formation and obstacle avoidance method also includes the strategy trajectory τ i Demonstrate trajectory with experts iE For comparison, if the strategy trajectory τ i Better than expert demonstration trajectory τ iE , then τ i Stored in temporary trajectory pool τ iT Otherwise, proceed to the next round; When the temporary trajectory pool τ iT When the storage limit is reached, the current expert demonstration trajectory τ iE Replaced by temporary trajectory pool τ iT , and clear the temporary trajectory pool τ iT .

2. The method according to claim 1, characterized in that The methods for training each AUV in formation include: Training the strategy model of the first AUV, which is used as the pilot AUV, is trained according to the strategy model training method of the AUV until convergence and the strategy model is saved; Train the strategy models of other AUVs except the first AUV, load the strategy models of all the AUVs in front of it, add the distance and angle between the AUV and the AUV in front and adjacent to it to the local observation of the AUV, train according to the strategy model training method of the AUV until convergence and save its strategy model.

3. The method according to claim 2, characterized in that When training the policy models of other AUVs except the first AUV, the policy network of the AUV is obtained by using Nash equilibrium.

4. The method according to claim 1, characterized in that: Discriminator In the step, the loss function is: in, Indicates that all state-action pairs (o, a) are real data expectations, Indicates that all data is generated data expectations.

5. The method according to claim 1, characterized in that Discriminator The method to output rewards is: Representation Discriminator According to the state action pair (o i,t ,a i,t ) is the reward at time step t.

6. The method according to claim 5, characterized in that Policy Network and value network In the update step, the IPPO algorithm is used for update, and the total loss function is expressed as: in, represents the loss function of updating the policy network, represents the loss function for updating the value network, represents the regularization of the policy network entropy, and c1 and c2 are proportional coefficients.

7. The method according to claim 1, characterized in that The AUV's strategy model makes decisions based on a partially observable Markov decision process.

Images