Multi-auv formation and obstacle avoidance method based on generative adversarial interactive self imitation learning

Abstract

The application discloses a multi-AUV formation and obstacle avoidance method based on generative adversarial interactive self imitation learning. The multi-AUV formation and obstacle avoidance method is based on multiple AUVs, each of which is respectively provided with a strategy network, a value network and a discriminator, and each of which also has a temporary trajectory pool. Each AUV has an expert demonstration trajectory. The multi-AUV formation and obstacle avoidance method comprises: formation training of each AUV, including: formation of all AUVs, and training of the strategy model of each AUV in the formation order; and a strategy model training method of the AUV, comprising: placing the AUV in a target environment to move, obtaining local observations at time steps, generating a strategy trajectory, and extracting a certain number of state-action pairs from the strategy trajectory and the expert demonstration trajectory to train the discriminator of the AUV. In the multi-AUV strategy model training process, each AUV only makes decisions through its own local observations, greatly reducing the communication delay, data loss and other problems caused by the need to frequently exchange information.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to a method for multi-AUV (Autonomous Underwater Vehicle) formation and obstacle avoidance through generative adversarial interactive self-imitation learning. Background Technology

[0002] The Earth's surface is mostly covered by oceans, yet human understanding and utilization of marine resources are currently quite limited. With the increasing scarcity of exploitable land resources, the development and utilization of marine biological and mineral resources have gradually become strategic goals for various countries. AUVs (Autonomous Underwater Vehicles) have a large operating range, require minimal surface support, and require no human intervention, making them ideal for seabed exploration, identification, and search missions. However, due to the limited detection range and energy storage of a single AUV, coupled with the increasing complexity of underwater missions, especially large-scale operations, multiple AUVs are often required to collaborate on tasks that a single AUV cannot accomplish. Formation control and obstacle avoidance are core technologies that ensure the successful completion of complex underwater missions by multiple AUVs.

[0003] The continuous development of Multi-Agent Reinforcement Learning (MARL) has provided new solutions for improving the control of multiple AUVs in uncertain marine environments. In MARL tasks, each agent interacts with the environment, makes decisions based on the current state, and receives environmental rewards to update the agent's policy, thereby continuously improving the accuracy of the agent's decisions.

[0004] Imitation learning can be combined with other learning mechanisms to improve its learning efficiency. Based on the different learning mechanisms combined, imitation learning can be divided into three categories: behavioral cloning, inverse reinforcement learning, and generative adversarial imitation learning. Behavioral cloning (BC) establishes a direct mapping between the agent's action space and state space through expert demonstrations, similar to the mapping between feature values ​​and labels in supervised learning. The advantage of this method is its simple framework and ease of implementation; however, its training results require a large amount of expert demonstration data, have high requirements for the accuracy of expert demonstrations, and have weak generalization ability, making it difficult to extend the policy to unknown environments.

[0005] Inverse Reinforcement Learning (IRL) uses expert demonstrations as the optimal solution to deduce a reward function, and then uses this reward function to train the agent through reinforcement learning methods, thereby enabling the policy to approximate the expert demonstration's performance in the task. This method can obtain a specific reward function, allowing the trained policy to achieve good performance in other environments. However, most IRL methods typically require building a model to solve a series of planning or reinforcement learning problems within the inner loop, which severely limits the performance of IRL methods for robot control in large and complex tasks. Summary of the Invention

[0006] This invention addresses the technical problems of difficulty in collecting excellent demonstration trajectories and poor policy stability in the multi-agent learning process in the prior art. It proposes a multi-AUV formation and obstacle avoidance method based on generative adversarial interaction self-imitation learning, which can solve the above problems.

[0007] To achieve the above-mentioned objectives, the present invention employs the following technical solution:

[0008] A multi-AUV formation and obstacle avoidance method based on generative adversarial interaction self-imitation learning, wherein the multi-AUV formation and obstacle avoidance method is based on multiple AUVs, each AUV is deployed with a policy network, a value network and a discriminator, each AUV also has a temporary trajectory pool, and each AUV has an expert demonstration trajectory. The multi-AUV formation and obstacle avoidance method includes:

[0009] Formation training for each AUV includes: forming all AUVs into a formation and training the policy model for each AUV in the formation order.

[0010] AUV policy model training methods include:

[0011] The AUV is placed in the target environment and moved to obtain local observations at time step t. i，t , where f represents the AUV number;

[0012] Local observation o i，t The policy network input to the AUV Get action a i，t ;

[0013] Perform action a i，t , obtain local observations for the next time step t+1, until all actions in the current round are completed;

[0014] Generate a set of state-action pairs for the current turn, which is the policy trajectory τ. i :

[0015] τ i =[(o i，1 a i，1 ), (o i，2 a i，2 ), ..., (o i,t a i，t )];

[0016] From the strategy trajectory τ i Demonstrating the trajectory τ with experts iE A certain number of state-action pairs are extracted from the data to train the discriminator of the AUV.

[0017] This round's identifier All output rewards update the policy network of the AUV. and value network

[0018] The multi-AUV formation and obstacle avoidance method also includes the strategy trajectory τ i Demonstrating the trajectory τ with experts iE Compare, if the policy trajectory τ i Superior to expert demonstration trajectory τ iE Then τ i Stored in temporary trajectory pool τ iT If it passes, proceed to the next round;

[0019] When the temporary trajectory pool τ iT When the storage limit is reached, the current expert demonstration trajectory τ will be... iE Replace with temporary trajectory pool τ iT At the same time, clear the temporary trajectory pool τ iT .

[0020] Compared with the prior art, the advantages and positive effects of the present invention are:

[0021] The generative adversarial interactive self-imitation learning method for multi-AUV formation and obstacle avoidance of the present invention, in the training process of multi-AUV policy model, deploys a policy network, a value network and a discriminator for each AUV, and each AUV makes decisions only through its own local observations, which greatly reduces the communication delay and data loss caused by the need for frequent information exchange.

[0022] To improve AUV training performance beyond the limitations of expert demonstrations, each AUV has its own temporary trajectory pool. This pool is specifically designed to store high-quality trajectories generated during training. The evaluation criteria for storing these policy-generated trajectories in the temporary pool are determined by human experts. When the temporary trajectory pool reaches its storage limit, it indicates that a sufficient number of high-quality trajectories have been collected, all of which are superior to the current expert demonstration. At this point, the current expert demonstration temporary trajectory pool is replaced with the new temporary trajectory pool, and the pool is cleared before moving on to the next stage of high-quality trajectory collection. This continuous replacement of expert demonstrations improves AUV training performance.

[0023] Other features and advantages of the present invention will become clearer after reading the detailed embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 The flowchart shows an embodiment of the present invention for generating adversarial interactive self-imitation learning-based multi-AUV formation and obstacle avoidance method.

[0026] Figure 2 This is a task diagram of one embodiment of the method of the present invention.

[0027] Figure 3 This is a schematic diagram of the detection range of the sonar sensor in one embodiment of the method of the present invention;

[0028] Figure 4 This is a schematic diagram of five actions involved in deploying the navigator AUV and the follower AUV in one embodiment of the method of the present invention;

[0029] Figure 5a This is a schematic diagram of the first obstacle avoidance task route provided in one embodiment of the method of the present invention;

[0030] Figure 5b This is a schematic diagram of the second obstacle avoidance route task provided in one embodiment of the method of the present invention;

[0031] Figure 5c This is a schematic diagram of the third obstacle avoidance route task provided in one embodiment of the method of the present invention;

[0032] Figure 6 This is a route comparison diagram of the navigator AUV and the follower AUV in one embodiment of the method of the present invention;

[0033] Figure 7 This is a route comparison diagram of the navigator AUV and the follower AUV in another embodiment of the method of the present invention;

[0034] Figure 8 This is a learning curve and confidence interval diagram of the Navigator AUV in various experiments in one embodiment of the method of the present invention;

[0035] Figure 9 This is a diagram showing the learning curves and confidence intervals of the navigator AUV and follower AUV in various experiments in one embodiment of the method of the present invention;

[0036] Figure 10 This is a schematic diagram of the trajectories generated by two AUVs in the first obstacle avoidance mission route in one embodiment of the method of the present invention;

[0037] Figure 11 A schematic diagram of the trajectories generated by two AUVs in a second obstacle avoidance mission route in one embodiment of the method of the present invention;

[0038] Figure 12 A schematic diagram of the trajectories generated by two AUVs in a third obstacle avoidance mission route in one embodiment of the method of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Example 1: This example proposes a multi-AUV formation and obstacle avoidance method based on generative adversarial interaction self-imitation learning. The method utilizes multiple AUVs, each deployed with a policy network, a value network, and a discriminator. Each AUV also has a temporary trajectory pool and expert demonstration trajectories. The multi-AUV formation and obstacle avoidance method includes:

[0041] Formation training for each AUV includes: forming all AUVs into a formation and training the policy model for each AUV in the formation order.

[0042] AUV policy model training methods include:

[0043] The AUV is placed in the target environment and moved to obtain local observations at time step t. it, where i represents the AUV number.

[0044] Local observation o i，t Policy network input to AUV Get action a i，t .

[0045] Perform action a i，t It obtains local observations at the next time step t+1 until all actions in the current round are completed.

[0046] Generate a set of state-action pairs for the current turn, which is the policy trajectory τ. i :

[0047] τ i =[(o i，1 a i，1 ), (o i，2 a i，2 ), ..., (o i,t a i,t )).

[0048] From the strategy trajectory τ i Demonstrating the trajectory τ with experts iE Extract a certain number of state-action pairs to train the discriminator of the AUV.

[0049] This round's identifier The policy network that updates all output rewards to AUV and value network

[0050] Multi-AUV formation and obstacle avoidance methods also include strategic trajectories τ i Demonstrating the trajectory τi with experts iE Compare, if the policy trajectory τ i Superior to expert demonstration trajectory τ iE Then τ i Stored in temporary trajectory pool τ iT If you pass, proceed to the next round; otherwise, you will advance to the next round.

[0051] When the temporary trajectory pool τ iT When the storage limit is reached, the current expert demonstration trajectory τ will be... iE Replace with temporary trajectory pool τ iT At the same time, clear the temporary trajectory pool τ iT .

[0052] This embodiment's generative adversarial interactive self-imitation learning method for multi-AUV formation and obstacle avoidance, in order to improve AUV training performance beyond the limitations of expert demonstrations, each AUV has its own temporary trajectory pool. This pool is specifically used to store superior trajectories generated during training. The evaluation criteria for storing these policy-generated trajectories in the temporary trajectory pool are provided by human experts. When the temporary trajectory pool reaches its storage limit, it indicates that a certain number of superior trajectories have been collected, and these trajectories are all better than the current stage's expert demonstrations. At this point, the current expert demonstration temporary trajectory pool is replaced with the current temporary trajectory pool, and the pool is cleared, initiating the next stage of superior trajectory collection. This continuous replacement of expert demonstrations improves AUV training performance.

[0053] Each AUV has its own independent policy network π i And it can be determined by the current environmental state s t Select action a i,t ~π i (a i |s=s t Each of them will receive their respective rewards r. i，t The ultimate goal of multi-AUV reinforcement learning is to enable each AUV to maximize its respective cumulative discounted reward.

[0054] Based on the characteristics of multi-AUV platooning and obstacle avoidance tasks and the underwater environment, this embodiment employs a decentralized training with decentralized execution (DTDE) communication framework. Under this framework, each agent is assigned to a policy network and a value network, independent of a central controller. Within this framework, each AUV is assigned to a policy network π(a i |o i ;θ i ) and a value network v(o i ;ω i During training, the i-th agent uses its equipped sensors to obtain local observations at the current time step t. i，t Then, the local observations are input into the policy network and the value network respectively to obtain the action decision 'a' for the current step. i，t And status evaluation. Each AUV makes decisions based solely on its own local observations, greatly reducing communication delays and data loss caused by frequent information exchanges.

[0055] In some embodiments, the method for training each AUV in formation includes:

[0056] Train the policy model of the first AUV, which will serve as the navigator AUV (hereinafter referred to as the navigator AUV). Train it according to the AUV policy model training method until convergence and save its policy model.

[0057] Train the policy models of all AUVs except the first AUV (hereinafter referred to as follower AUVs), load the policy models of all the AUVs that are ahead of it, add the distance and angle between the AUV and the AUV in front of and adjacent to it to the local observation of the AUV, train until convergence according to the AUV policy model training method, and save its policy model.

[0058] In the method for training each AUV in formation, firstly, the strategy model of the first AUV, AUV1 (the simulation model of the lead AUV), is loaded into the marine environment and trained until convergence, then the strategy model is saved. Next, the strategy model of the second AUV, AUV2, is loaded. Based on the already trained strategy model, the simulation model of AUV2 is added, and AUV2 is trained further. The performance of AUV2's strategy π2 is influenced by state information such as distance and angle with AUV1. Similarly, based on the strategy models of AUV1, AUV2…AUVi-1, the simulation model of AUVi is added, and AUVi is trained further, with the strategy π2 of the i-th AUV being trained. i Its performance will be affected by other AUV strategies.

[0059] In some embodiments, when training the policy models of AUVs other than the first AUV, Nash equilibrium is used to obtain the policy network of the AUVs.

[0060] In complex training relationships involving multiple AUVs, the most common outcome is Nash equilibrium (NE), which can be defined as:

[0061]

[0062] Under Nash equilibrium, if other AUVs do not change their policies, and the i-th AUV cannot obtain a higher cumulative discounted return no matter how it changes its policy, then the policy π of that AUV is considered to be... i The optimal strategy under the current circumstances has been achieved. The effect is that, in the process of multi-AUV formation and obstacle avoidance, the learning process of each AUV needs to reach a dynamic equilibrium.

[0063] like Figure 1 The flowchart shown is a method for generating adversarial interactive self-imitation learning for multi-AUV formation and obstacle avoidance provided by an embodiment of the present invention.

[0064] Specifically, the input is expert demonstrations of each AUV.iE ~π iE Randomly initialize the policy network, value network, and discriminator's policy network parameters θ. i , and value network parameters φ i Initialize the temporary trajectory pool τ iT The capacity is N. During the iteration process, each AUV uses its own sensor to obtain local observations at the current time step. i,t Each AUV utilization strategy network Based on their respective local observations at the current step i，t Select and execute action a i,t ; The state-action pairs generated by each AUV at the current time step (o i,t a i，t Store it in the corresponding strategy trajectory set τ i In the middle; human experts assessed the strategy trajectory τ for this round. i Add to temporary trajectory pool τ? iT If the temporary trajectory pool τ iT When the storage limit N is reached, the temporary trajectory pool τ will be... iT Data coverage extends to expert demonstrations. iE And initialize the temporary trajectory pool τ iT Update the parameters ω of each discriminator using the ADAM optimizer. i ; Utilizing each discriminator The reward for each AUV in this round is obtained based on the strategy trajectory.

[0065] During training, the IPP0 algorithm is used to learn a decentralized policy network. Simultaneously utilizing local observations i,t And GAE to train a value network Vi. From the policy trajectory τi and the current expert demonstration τ, respectively. iE A certain number of state-action pairs are extracted from the data to train the discriminator for the i-th AUV.

[0066] In some embodiments, the discriminator In this step, the loss function is:

[0067]

[0068] in, When all state-action pairs (o, a) are real data. Expectations When indicating that all data is generated data The expectation.

[0069] The discriminator of the i-th AUV It will be based on the state and action pairs in all strategy trajectories of this round.

[0070] In some embodiments, the discriminator The method for outputting rewards is as follows:

[0071]

[0072] Discriminator Based on the state action pair (o i,t a i,t The reward at time step t.

[0073] All rewards from the discriminator output of the i-th AUV in this round will be used to train the policy network. Value Network V i .

[0074] In some embodiments, the policy network Value Network V i In the update step, the IPPO algorithm is used for updating, and the total loss function is expressed as:

[0075]

[0076] in, This represents the loss function of the policy update network. This represents the loss function used to update the value network. c1 and c2 represent the regularization of the policy network entropy, and c1 and c2 are scaling factors.

[0077] To address the multi-task problem, a method for extracting expert trajectories through classification is proposed to ensure that the discriminator can provide accurate reward values. First, each AUV will receive the trajectory τ demonstrated by an expert. iE During each training round, the i-th AUV uses its own sensors to obtain local observations at the current time step t. i，t Input local observations into the policy network Action a can be obtained i，t After executing the selected action, the i-th AUV will obtain the observation at the next time step t+1, and this cycle repeats until the end of time. After t arbitrary time steps, the set of all state-action pairs generated by the i-th AUV is produced:

[0078] τ i =[(o i，1 a i，1 ), (o i，2 a i，2 ), ..., (o i,t a i,t )).

[0079] From expert demonstration τ i ~π i and policy trajectory τ iE ~π iE Extract a certain number of state-action pairs to train the discriminator for the i-th AUV. All rewards from the discriminator output of the i-th AUV in this round will be used to train the policy network. and value network

[0080] In some embodiments, the strategy model of the AUV makes decisions based on a partially observable Markov decision process.

[0081] This embodiment utilizes a partially observable Markov decision process (POMDP). For reinforcement learning in environments where agents can acquire reliable state signals, Markov decision process (MDP)-based methods have achieved considerable success. However, in practical applications, the limited sensor capabilities of agents often prevent accurate acquisition of global observation information. In such cases, POMDPs allow agents to make reliable decisions even with incomplete observations. As an extension of POMDPs, decentralized partially observable Markov decision processes (Dec-POMDPs) model multiple agents under the same environmental conditions, allowing them to make decisions independently with partial environmental observations to obtain the optimal cooperative strategy.

[0082] A decentralized partially observable Markov decision process can be defined as a tuple Here, n represents the number of agents in the environment. Global observations are represented by a finite state space array S.

[0083] A = ×iA i This represents the joint action space, where A i Let a represent the action space of the i-th agent. The joint action of all agents at time t can be represented as a. t = 1，t a 2，t , ..., a n,t Each agent can only obtain its own action value.

[0084] T:S×A×S→P​T (S)∈[0,1] represents the state transition function, which describes the random influence of joint actions on the global state space. Therefore, the transition probability of the next global observation can be obtained from the current global observation and joint actions, i.e.:

[0085]

[0086] O = × i O i Then it represents the set of joint local observations, O i Let represent the local observation space of the i-th agent. The joint local observations obtained by all agents during their interaction with the environment at time t can be represented as o. t = <o 1，t o 2，t , ..., o n,t Each agent can only obtain its own local observations.

[0087] f: A×S→Pf(0)∈[0,1] represents the observation function, describing how the agent extracts local observations from the global observation space. In practical applications, it represents the observations that can be obtained within the sensor's detection range, and is expressed as a probability distribution mapping from the current global observation to the joint local observation, i.e.:

[0088]

[0089] This represents the initial state distribution of each agent in the environment at time t=0.

[0090] This means that the joint reward function of the agents can be represented by the global observation and joint action at the current time step, and the global observation at the next time step:

[0091]

[0092] In distributed observable Markov decision-making, the ultimate goal is to maximize the joint policy π = <π1, π2, ..., π. n >any sub-policy π i The cumulative expected return from time t to time T is:

[0093]

[0094] Here, γ∈[0,1] is a decay factor, representing the correlation between the cumulative expected reward at the current moment and the reward obtained at future moments. The smaller γ is, the more the agent focuses on the reward obtained at the current moment; conversely, the larger γ is, the more the agent values ​​the performance at future moments.

[0095] Furthermore, in multi-agent reinforcement learning, multiple agents interact with the environment simultaneously. Each agent i possesses an independent policy network π. i And it can be determined by the current environmental state s t Select action a i,t ~π i (a i |s=s t Each of them will receive their respective rewards r. i，t The ultimate goal of multi-agent reinforcement learning is to enable each agent to maximize its respective cumulative discounted reward. However, in a multi-agent system, the policy π of the i-th agent... i The performance of an agent is influenced by the policies of other agents. In complex relationships involving multiple agents, the most common solution is the Nash Equilibrium (NE), which can be defined as: Under Nash equilibrium, if other agents do not change their policies, and the i-th agent cannot obtain a higher cumulative discounted reward no matter how it changes its policy, then the agent's policy π is considered to be... i The optimal strategy under the current circumstances has been achieved. The effect is that, in multi-agent reinforcement learning, the learning process of each agent needs to reach a dynamic equilibrium.

[0096] Combining all the above technical solutions, the advantages and positive effects of this embodiment are as follows: This solution combines a reinforcement learning framework for multi-AUV formation and obstacle avoidance, a communication framework for decentralized training and decentralized decision-making, and a generative adversarial self-imitation learning method incorporating human feedback to form a generative adversarial interactive self-imitation learning method for multi-AUV formation and obstacle avoidance. This invention uses an intelligent agent to leverage the synergy of demonstration and human evaluation, referencing both expert demonstrations and human knowledge, thus overcoming the limitations of demonstration quality and achieving learning results surpassing expert demonstrations. This effectively learns better strategies, improves strategy stability, and reduces the cost of collecting expert demonstrations in practical applications.

[0097] The technical effects of the present invention will be described in detail below with reference to experiments.

[0098] To verify the feasibility of the MAGAISIL algorithm in multi-AUV formation and obstacle avoidance control, this experiment deployed a reasonable state space and action space for the multi-AUV system. Furthermore, to visualize the training effect, a simple reward function was designed as an evaluation criterion, and expert trajectories of different levels were specified through the reward function. We designed three tasks of varying difficulty, and the experimental environment was mainly built using the ROS robot operating platform and Gazebo simulation software. Simultaneously, we used UUVSimulator to simulate the deployment of the multi-AUV system and sonar sensors.

[0099] (I) State Space Setup

[0100] Figure 2 This is a schematic diagram of a multi-AUV formation and obstacle avoidance mission. The black squares in the diagram represent obstacles, each 0.5 meters wide. O This represents the distance between obstacles, set to 30 meters in the mission; (x L y L (x) represents the current position coordinates of the navigator AUV; F y F The coordinates of the follower AUV are shown in the image. The blue and green sectors represent the sonar sensor detection ranges of the navigator and follower AUVs, respectively. For the follower AUV, in addition to maintaining distance from obstacles, it also needs to maintain a certain distance from the navigator AUV. Therefore, its observation can be divided into two parts: the observation of tracking the navigator AUV. and observation of the obstacle detection part of sonar

[0101] Regarding the formation portion, for the follower AUV, the observation of its tracking of the navigator AUV can be represented as: g F The straight-line distance between the follower AUV and the navigator AUV is represented by the following formula:

[0102]

[0103] a F This indicates the heading deviation of the follower AUV, i.e., its current heading a. FH The angle between the current position and the position of the navigator AUV can be represented as follows:

[0104]

[0105] Regarding obstacle avoidance, this experiment uses sonar sensors to detect the position of obstacles in real time, enabling the AUV to maintain a safe distance from obstacles. The sonar angle range was set to... The detection range was set to 33 meters, and the detection beam was set to 600 beams. This experiment uses a sonar sensor to detect the location of obstacles in real time, enabling the AUV to maintain a safe distance from obstacles. The sonar angle range was set to... The detection range was set to 25 meters, and the detection beam was set to 600 beams. To ensure that each AUV could accurately obtain the position and angle of obstacles, the sonar information was processed accordingly.

[0106] like Figure 3 As shown, the sonar sensor has an opening angle of 120°, dividing the total detection area into six small sector intervals, each occupying 20°. These sector intervals are [-60°, -40°], [-40°, -20°], [-20°, 0°], [0°, 20°], [20°, 40°], and [40°, 60°]. By inputting the shortest distance to the obstacle detected in each small region as an observation vector into the observation space, the AUV can accurately determine the current location and angle of the obstacle, thus enabling it to avoid it accurately. For the Navigator AUV, its sonar obstacle detection observation constitutes its observation space. L , can be represented as

[0107] o L ={d L1 d L2 d L3 d L4 d L5 d L6}#

[0108] For a follower AUV, the observation space is obtained by merging the observations from the tracking AUV portion with the sonar obstacle detection portion. F , can be represented as

[0109] o F ={g F a F d F1 d F2 d F3 d F4 d F5 d F6}#

[0110] All vectors in the two AUV observation spaces are normalized to the range of [-1, 1] to ensure that the input values ​​are within a reasonable range and can be mapped to the non-saturation range of the nonlinear activation function of the neural network, thereby ensuring the stability of training.

[0111] Motion space settings

[0112] The experimental task in this chapter still requires the AUV to operate on a two-dimensional plane, so the rudder blades 2 and 4, which control the AUV's ascent and descent, are still set to 0. For example... Figure 4 As shown, the navigator AUV and follower AUV have the same motion space setup, with each AUV equipped with five maneuvers: two left turns at different angles, one straight turn, and two right turns at different angles. The propellers for these five maneuvers operate at 300 revolutions per minute.

[0113] Task settings

[0114] In the experiments of this chapter, three different obstacle avoidance tasks were designed. Task 1 was used to train the algorithm, while Tasks 2 and 3 were used to test the algorithm's adaptability. It should be noted that neither the navigator AUV nor the follower AUV pre-obtained the obstacle's position information before executing the task. Instead, they adjusted their position based on information obtained from sonar sensors during the task to maintain a safe distance from the obstacle.

[0115] Task 1

[0116] The setup for Task 1 is shown in Figure 5(a). The gray lines represent obstacles 0.5 meters wide, positioned between (0,30) and (60,30), (0,0) and (90,0), (60,30) and (60,110), (90,0) and (90,80), (60,110) and (180,110), (90,80) and (150,80), (180,110) and (180,30), (150,80) and (150,0), (180,30) and (240,30), with a distance of 30 meters between obstacles. Since the multi-AUV formation and obstacle avoidance task is implemented on a two-dimensional plane, the obstacles must be positioned at appropriate heights to ensure accurate detection by both AUVs. In the diagram, the blue dot at (1,0) and the red dot at (18,0) represent the starting points of the navigator AUV and the follower AUV, respectively. In this mission, the navigator AUV needs to remain in the middle of the obstacles on both sides while navigating the obstacle zone, while the follower AUV needs to follow the navigator AUV 18 meters behind it while maintaining a safe distance from the obstacles on both sides.

[0117] To ensure the feasibility of the experiment and avoid the two AUVs engaging in excessive and meaningless exploration, the mission will be considered a failure if any of the following situations occur during the experiment:

[0118] The Navigator AUV is too close to the obstacle, i.e., |d L |≤2.

[0119] The follower AUV is too close to the obstacle, i.e., |d L |≤2.

[0120] The follower AUV maintains a distance that is either too close or too far when tracking the leader AUV, i.e., |a L |<3 or|a F |>33.

[0121] The follower AUV's heading deviation was too large, i.e.

[0122] Once one of the two AUVs meets the above conditions, the mission for that round will terminate, and both AUVs will be reset to their initial positions to begin the next round of mission.

[0123] Task 2

[0124] To test whether the trained model could successfully complete the task in a more complex setting, we added more obstacles to Task 1, which is called Task 2. The setup is shown in Figure 5(b). Rectangular obstacles with a length of 5 meters and a width of 20 meters were placed at (87.5,40), (62.5,70), (177.5,70), and (152.5,40), and rectangular obstacles with a length of 20 meters and a width of 5 meters were placed at (100,82.5), (120,107.5), and (140,82.5). The purpose of Task 2 was to add obstacles to create additional interference and explore the model's adaptability.

[0125] Task 3

[0126] Task 3 changed the placement angle of the boundary obstacles and increased the total length of the obstacle area, as shown in Figure 5(c). In Task 3, obstacles with a width of 0.5 meters were placed between (0,30) and (60,30), (0,0) and (70,0), (60,30) and (110,80), (70,0) and (120,50), (110,80) and (130,80), (120,50) and (170,0), (130,80) and (180,30), (170,0) and (190,0), (180,30) and (230,80), (190,0) and (240,50), (230,80) and (300,80), respectively. The purpose of Task 3 is to test whether the model can successfully complete formation and obstacle avoidance tasks in obstacle areas with different angles and in longer tasks.

[0127] Reward function design

[0128] During the training of the MAGAISIL algorithm, the rewards output by the discriminators of the two AUVs can only be used as training data and cannot be used to evaluate the performance of the AUVs in the task. Therefore, this section designs a simple reward function for the two AUVs according to the task requirements, but it is not used for AUV training; it is only used as an evaluation criterion for AUV performance.

[0129] Navigator AUV's reward design

[0130] In a multi-AUV formation and obstacle avoidance mission, the navigator AUV needs to maintain its position between obstacles on both sides as it traverses the mission area. The distance d between the obstacles on both sides is known in Mission 1. O The distance is 15 meters, while the opening angle of the AUV sonar sensor is... After conversion, it can be seen that when the shortest distance d between the AUV and the obstacle... L Maintaining a distance of approximately 17.3 meters ensures that the Navigator AUV remains in the center of the obstacle, thus its reward function r L It can be defined as:

[0131]

[0132] Reward design for follower AUVs

[0133] Unlike the navigator AUV, the follower AUV in multi-AUV formation and obstacle avoidance missions needs to simultaneously perform the following tasks: tracking the navigator AUV and maintaining a safe distance from obstacles on both sides. Therefore, its reward function r F It can be divided into two parts:

[0134]

[0135] in, This represents the reward for the follower AUV tracking the navigator AUV portion. This represents the reward for the obstacle avoidance portion of the follower AUV. Since the follower AUV needs to perform two tasks simultaneously, the reward function cannot be biased towards one side. Therefore, each part of the reward is multiplied by 0.5 as a balancing coefficient. It will be based on the straight-line distance g between the follower AUV and the navigator AUV F deviation from heading a F Providing rewards can be represented as:

[0136]

[0137] It can be observed that the greater the straight-line distance between the follower AUV and the navigator AUV (beyond 18 meters) or the larger the course deviation, the smaller this reward becomes; conversely, the smaller the distance, the larger the reward. It is the shortest distance d from the obstacle detected by its sonar sensors. F It was determined, after conversion, that the shortest distance d between the AUV and the obstacle... F Maintaining a distance of approximately 17.3 meters ensures that the follower AUV remains in the middle of the obstacle, which can be represented as:

[0138]

[0139] As can be seen, the greater the deviation of the follower AUV from the center of the obstacle, the smaller the reward for that part; conversely, the smaller the deviation, the greater the reward. Similarly, in the missions of this chapter, even if the follower AUV triggers the mission termination condition, it will not give a large negative reward value.

[0140] Expert trajectory selection

[0141] Second-best expert demonstration acquisition

[0142] In training the leader AUV and follower AUV using the IPPO algorithm with environmental rewards, if in a certain round the leader AUV's cumulative reward reaches approximately 335 while the follower AUV also obtains a cumulative reward of around 355, then the trajectory generated by the two AUVs in that round is considered a suboptimal expert demonstration. We selected four rounds of such trajectories as suboptimal expert demonstrations, such as... Figure 6 As shown.

[0143] As can be seen from the trajectory diagram, in these four rounds, while the navigator AUV could successfully pass through the obstacle area, it could not guarantee that it would maintain its position between the obstacles on both sides. Meanwhile, although the follower AUV could meet the mission requirements of tracking the navigator AUV and passing through the obstacle area, its tracking error was significantly larger.

[0144] Optimal expert demonstration acquisition

[0145] If, in a single round, both the leader AUV and the follower AUV accumulate approximately 400 points in rewards, then the trajectory generated by the two AUVs in that round is considered to have reached the optimal expert demonstration level. Similar to the suboptimal expert demonstration, we also selected four rounds of this trajectory as the optimal expert demonstration, such as... Figure 7 As shown.

[0146] Experimental Results Analysis

[0147] To facilitate interaction between human experts and AUVs in collecting optimal trajectories, this chapter's experiments train two AUVs separately. For the navigator AUV, the following three experiments were conducted:

[0148] The suboptimal expert demonstration was input into the Navigator AUV and trained for 1000 rounds using the MAGAISIL algorithm in Task 1. The trained model was then saved.

[0149] Input the best expert demonstration into the Navigator AUV and train it for 1000 rounds using the MAGAIL algorithm in Task 1, then save the trained model.

[0150] The suboptimal expert demonstration was input into the Navigator AUV and trained for 1000 rounds using the MAGAIL algorithm in Task 1.

[0151] Repeat each experiment three times, and save the cumulative reward for each round of each experiment to obtain the learning curve and confidence interval of the Navigator AUV in each experiment, such as... Figure 8 As shown.

[0152] As shown in the graph, the Navigator AUV trained using the MAGAIL algorithm with optimal expert demonstrations reaches its best performance around round 230, with a stable cumulative reward of around 400 per round. The Navigator AUV trained using the MAGAIL algorithm with suboptimal expert demonstrations experiences cumulative rewards that fluctuate around the suboptimal expert demonstration level but do not reach the level of optimal expert demonstrations. For the MAGAISIL algorithm, the reward curve before 150 rounds is very similar to that generated by the MAGAISIL algorithm based on suboptimal expert demonstrations. This is because at the beginning of training, the expert demonstration pool of the MAGAISIL algorithm has not been replaced, and it still uses suboptimal expert demonstrations for learning. After 150 rounds, the reward curve generated by the MAGAISIL algorithm gradually increases. This is because human experts judge that there are policy trajectories superior to suboptimal expert demonstrations during this period and store them in a temporary trajectory pool. When the trajectory pool reaches its limit, it replaces the current expert demonstration to guide the Navigator AUV to learn better expert demonstrations. At about 450 rounds, after several rounds of replacement with better trajectories, the MAGAISIL algorithm can reach the optimal expert demonstration level, and the cumulative reward remains around 400 in each subsequent training round. By comparison, it can be found that, under the premise of using suboptimal expert demonstrations, the MAGAISIL algorithm can continuously collect the excellent trajectories generated by the policy, guide the Navigator AUV's training level to continuously improve, and thus achieve the effect of optimal expert demonstrations.

[0153] For follower AUVs, the following three experiments were conducted:

[0154] Load the navigator AUV model trained with the MAGAISIL algorithm based on the suboptimal expert demonstration, and input the suboptimal expert demonstration into the follower AUV to train it for 1000 rounds in Task 1 using the MAGAISIL algorithm. Then save the trained model.

[0155] Load the navigator AUV model trained using the MAGAIL algorithm based on the best expert demonstration, input the best expert demonstration into the follower AUV, and train it for 1000 rounds using the MAGAIL algorithm in Task 1. Then save the trained model.

[0156] Load the Navigator AUV model trained using the MAGAIL algorithm based on the best demonstration, input the suboptimal expert demonstration into the Navigator AUV, and train it for 1000 rounds using the MAGAIL algorithm in Task 1.

[0157] Repeat each experiment three times, and save the cumulative reward for each round of each experiment to obtain the learning curve and confidence interval of the follower AUV in each experiment, such as... Figure 9 As shown.

[0158] As shown in the figure, the reward curve of the follower AUV trained by the MAGAIL algorithm based on suboptimal demonstrations consistently rises before round 200. Around round 200, it reaches a level similar to the suboptimal demonstration and remains stable, unable to surpass it. The reward curve of the follower AUV trained by the MAGAIL algorithm based on optimal demonstrations converges faster, reaching the optimal demonstration level and remaining stable within about 100 rounds. Between approximately 180 and 380 rounds, with the replacement of expert demonstrations, the training performance of the follower AUV gradually improves, surpassing the suboptimal expert demonstration. Around 400 rounds, the training effect of the MAGAIL algorithm reaches a level similar to that of MAGAIL based on optimal demonstrations and remains stable. Through the comparison of reward curves, it can be demonstrated that the MAGAISIL algorithm, based on suboptimal expert demonstrations, can gradually collect excellent trajectories generated by the strategy through human expert evaluation, thereby surpassing the suboptimal expert demonstration level and achieving an effect similar to the optimal demonstration.

[0159] Model Adaptability Analysis

[0160] During the training process of the MAGAISIL algorithm based on suboptimal demonstration and the MAGAIL algorithm based on optimal demonstration in Task 1, the trained policy model is saved and loaded into Task 1, Task 2 and Task 3 respectively for testing. The test data of each round is saved and analyzed.

[0161] Model performance in Task 1

[0162] The trained leader and follower AUV models were loaded into Task 1 for testing. During the test, the trajectories generated by the two AUVs were as follows: Figure 10As shown. For the navigator AUV, the trajectory generated by the model trained using the MAGAISIL algorithm based on the suboptimal demonstration is close to that of the MAGAIL algorithm based on the optimal demonstration. In the task test, it can maintain its position between the two boundary obstacles and pass through the obstacle area. For the follower AUV, although the model trained using the MAGAIL algorithm based on the optimal demonstration can successfully track the navigator AUV and pass through the obstacle area, the tracking effect changes significantly when the angle of the boundary obstacles changes, and it cannot well balance the tracking and obstacle avoidance tasks.

[0163] Analyzing the state information generated by the models trained by the two algorithms in Task 1, it was found that neither the suboptimal expert demonstration of the navigator AUV nor the follower AUV could complete the task with high accuracy in multi-AUV formation and obstacle avoidance. However, after training with the MAGAISIL algorithm using the suboptimal expert demonstration, the navigator AUV could maintain a distance of about 15 meters from obstacles, i.e., the middle of the obstacles on both sides, when the obstacle angle did not change. Although it experienced significant fluctuations at turns, it quickly adjusted to maintain its safe position.

[0164] After being trained using the MAGAISIL algorithm with suboptimal expert demonstration, the follower AUV can maintain a distance of about 18 meters from the navigator AUV when no turns are required, while ensuring a safe distance of about 15 meters from boundary obstacles and a heading deviation of about 0 rad. After fluctuations occur during turns, it can quickly adjust its own position to return to normal formation and obstacle avoidance tasks.

[0165] In the test of Task 1, the two AUV models trained by the MAGAISIL algorithm based on suboptimal expert demonstrations achieved training results that were similar to those of the MAGAIL algorithm based on optimal demonstrations.

[0166] Model performance in Task 2

[0167] To test the algorithm's adaptability, the two trained AUV models were loaded into Task 2 for testing. The task trajectories generated during the test are as follows: Figure 11As shown, the model trained by the MAGAISIL algorithm based on suboptimal expert demonstrations, even with the addition of extra obstacles to Task 1, can still ensure that both AUVs maintain a safe distance from obstacles while passing through the obstacle area and successfully completing the formation task. Furthermore, the final training performance of the MAGAISIL algorithm based on suboptimal expert demonstrations is comparable to that of the MAGAISIL algorithm based on optimal expert demonstrations. In addition, for the follower AUV, the model trained by the MAGAISIL algorithm can better adjust to a suitable position to track the leader AUV after avoiding additional obstacles.

[0168] The state information generated by the models trained by the two algorithms in Task 2 was analyzed. In Task 2, the Navigator AUV models trained by both algorithms were able to maintain a distance of approximately 15 meters between the boundary obstacles on both sides when turning and obstacle avoidance were not required. After turning, they were able to adjust their position in a timely manner to maintain a safe distance until they passed through the obstacle area. The MAGAISIL algorithm can utilize suboptimal expert demonstrations and continuously replace the demonstration data with human experts, enabling the trained Navigator AUV to achieve a level similar to that of the MAGAIL algorithm based on optimal demonstrations.

[0169] In Task 2, the addition of extra obstacles increased the tracking error of the follower AUV. The model trained using the MAGAISIL algorithm, through trajectory replacement, achieved an effect similar to the optimal expert demonstration, thus overcoming the low accuracy problem of suboptimal expert demonstrations to some extent. Analysis of the state information reveals that the heading deviation and distance information between the model trained by the MAGAISIL algorithm in Task 2 are very close to those of the MAGAISIL algorithm based on the optimal expert demonstration; however, the distance to the navigator AUV differs significantly. This difference may be due to the limited number of rounds restricting the number of expert demonstration replacements; while the replaced demonstrations are close to optimal, they do not reach the truly optimal level. Overall, the model trained by the MAGAISIL algorithm adapts well to environments with more obstacles and completes the task at a high level.

[0170] Model performance in Task 3

[0171] To test whether the model can successfully complete formation and obstacle avoidance tasks over long distances, the trained model was loaded into three different tasks for testing. The placement angles of boundary obstacles were varied to discuss the model's adaptability to different environments. The resulting task trajectories during the tests are shown below. Figure 12As shown in the figure, similar to the results of the previous two experiments, the trajectory generated by the leader AUV model trained by the MAGAISIL algorithm based on suboptimal expert demonstrations is very close to that of the MAGAISIL algorithm based on optimal expert demonstrations. The generated trajectories are essentially identical, as can be seen from the figure. For the follower AUV, the model trained by the MAGAISIL algorithm can better balance formation and obstacle avoidance tasks at turns, tracking the leader AUV with less deviation while maintaining a safe distance from boundary obstacles.

[0172] Analyzing the state information generated by the models trained by the two algorithms in Task 3 within one round, for the Navigator AUV, the distance information between the model and the obstacles fluctuates less at turns because the angle of the boundary obstacles is set smaller in Task 3. When turning is not required, the Navigator AUV model trained by the MAGAISIL algorithm can maintain its position between the obstacles on both sides, and can also quickly adjust its position to maintain a safe distance from the boundary obstacles after turning. As can be seen from the figure, the Navigator AUV model trained by the MAGAISIL algorithm based on suboptimal expert demonstrations can achieve a very similar level to the MAGAISIL algorithm based on optimal expert demonstrations.

[0173] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions claimed by the present invention.

Claims

1. A multi-AUV formation and obstacle avoidance method of generative adversarial interactive self-imitation learning, characterized in that, The multi-AUV formation and obstacle avoidance method is based on multiple AUVs, each of which is respectively deployed with a strategy network, a value network and a discriminator, and each of which has a temporary trajectory pool, and each of which has an expert demonstration trajectory, and the multi-AUV formation and obstacle avoidance method comprises: The formation training of each AUV comprises: The strategy model training method of the AUV comprises: placing the AUV in motion in the target environment to obtain a local observation o at time step t i,t where i denotes the number of the AUV; obtain local observations o i,t input to the policy network of the AUV obtain action a i,t ; Perform action a i,t Obtain local observations for the next time step t+1 until all actions for the current episode are completed. generating a collection of state-action pairs for the current round, for a policy trajectory τ i : τ i = [(o i,1 ,a i,1 ), (o i,2 ,a i,2 ),…,(o i,t ,a i,t )] ; extracting a number of state-action pairs from a policy trajectory τ i and an expert demonstration trajectory τ iE training a discriminator for the AUV from the number of state-action pairs The round discriminator All rewards output update the policy network of the AUV and value network The multi-AUV formation and obstacle avoidance method further comprises comparing the strategy trajectory τ i with the expert demonstration trajectory τ iE , if the strategy trajectory τ i is better than the expert demonstration trajectory τ iE , then storing τ i into a temporary trajectory pool τ iT , otherwise entering the next round; When the temporary trajectory pool τ iT reaches a storage upper limit, the current expert demonstration trajectory τ iE is replaced by the temporary trajectory pool τ iT , and the temporary trajectory pool τ iT is emptied at the same time.

2. The method of claim 1, wherein, The method for training each AUV in the formation comprises: The strategy model of the first AUV is trained as a leading AUV, and the strategy model of the AUV is trained to convergence according to the strategy model training method of the AUV and is saved; The strategy model of the AUV other than the first AUV is trained, the strategy model of all AUVs in front of the AUV is loaded, the distance and angle between the AUV and the AUV in front of and adjacent to the AUV are added to the local observation of the AUV, the strategy model of the AUV is trained to convergence according to the strategy model training method of the AUV, and the strategy model of the AUV is saved.

3. The method of claim 2, wherein, In the training of the strategy model of the AUV other than the first AUV, Nash equilibrium is used to obtain the strategy network of the AUV.

4. The method of claim 1, wherein, Discriminator In the step, the loss function is: where represents the expectation of when all state-action pairs (o, a) are real data, represents the expectation of when all data are generated data.

5. The method of claim 1, wherein, Discriminator The method of outputting the reward is: representing a discriminator action according to state (o i,t ,a i,t ) reward at time step t.

6. The method of claim 5, wherein, Policy network and value network In the update step, the IPPO algorithm is used for updating, and the total loss function is represented as: wherein, represents a loss function for updating the policy network, represents a loss function for updating the value network, represents a regularization of the policy network entropy, and c1 and c2 are proportionality coefficients.

7. The method of claim 1, wherein, The strategy model of the AUV is based on a partially observable Markov decision process to make decisions.

Images