A method for generating formation and obstacle avoidance strategies for multiple autonomous underwater vehicles

Abstract

This invention provides a method for generating formation and obstacle avoidance strategies for multiple autonomous underwater vehicles (AUVs), belonging to the field of underwater vehicle obstacle avoidance control technology based on reinforcement learning. This invention incorporates a large language model optimized through cueing engineering into a multi-agent generative adversarial interactive self-imitation learning framework, replacing the role of a traditional human evaluator in the trajectory evaluation stage. This enables autonomous optimization of formation maintenance and obstacle avoidance strategies for multiple AUVs in complex underwater environments. Experimental results show that this framework can effectively guide the large language model, enabling it to autonomously and reliably evaluate trajectory quality, thereby significantly reducing expert supervision costs and training time while maintaining high performance. Compared with traditional interactive imitation learning methods that rely on human feedback, the trajectories generated by this invention exhibit smoother and more natural motion characteristics, effectively improving the overall stability and operational efficiency of multi-AUV systems in actual underwater missions.

Description

Technical Field

[0001] This invention belongs to the field of underwater vehicle obstacle avoidance control technology based on reinforcement learning, and particularly relates to a method for generating multi-autonomous underwater vehicle formation and obstacle avoidance strategies. Background Technology

[0002] As the complexity of ocean exploration missions increases, multi-AUV swarm control plays a crucial role in underwater search, resource exploration, and monitoring missions due to its high operational efficiency and stability. Maintaining stable swarm geometry and achieving efficient obstacle avoidance are core technological challenges in these missions.

[0003] Traditional reinforcement learning methods rely heavily on manually designed reward functions when solving multi-agent tasks. However, the underwater environment is highly nonlinear and uncertain, and performance metrics such as "formation smoothness" and "movement naturalness" are difficult to express precisely using explicit mathematical formulas. This often results in significant trial-and-error costs in designing reward functions.

[0004] Imitation learning trains policies by learning from expert demonstration data. Generative Adversarial Imitation Learning (GAIL), in particular, can handle high-dimensional continuous action spaces through adversarial training between a discriminator and a generator. However, the performance ceiling of GAIL is limited by the quality of the expert demonstration data. In practical underwater operations, obtaining consistently optimal expert demonstrations is often impractical and expensive.

[0005] Interactive imitation learning introduces human evaluators to score or provide feedback on the trajectories generated by the agent in real time, thereby continuously optimizing suboptimal initial data through interaction. However, this "human-in-the-loop" model has obvious "human bottlenecks": human experts are prone to fatigue and subjective bias during long-term, large-scale training, and manual annotation is extremely costly, limiting the convergence speed and scalability of the algorithm.

[0006] Therefore, how to automate the evaluation process using advanced artificial intelligence technology and replace human evaluators without sacrificing performance is a key issue that urgently needs to be addressed in the field of autonomous learning for multi-AUV systems. Summary of the Invention

[0007] To address the above problems, this invention provides a method for generating multi-autonomous underwater vehicle formation and obstacle avoidance strategies, comprising the following steps: S1, initialize the multi-AUV environment and the policy network, critic network, discriminator network and expert trajectory pool of each AUV agent, as well as a temporary demonstration buffer for storing candidate high-quality trajectories; S2, In the training round, each AUV agent generates and executes control actions based on the current perceived local environment state using the policy network, forming the motion trajectory data for the current round, and extracting the multimodal feature information of the trajectory; S3, Construct a large language model evaluator, which transforms the multimodal feature information into text descriptions and key indicator data that the large language model can parse through prompting engineering, and the large language model scores the trajectory quality. S4. The expert trajectory pool is dynamically updated based on the scoring results. First, the current trajectory is filtered, and the high-scoring trajectory is stored in a temporary demonstration buffer. When the preset conditions are met, the trajectory in the temporary demonstration buffer is used to update the expert trajectory pool. S5, sample state-action pairs from the updated expert trajectory pool and the trajectory generated by the current AUV respectively, train the discriminator network to distinguish between expert trajectories and agent trajectories, and generate a mimicry reward signal; S6. Combining the simulated reward signal, the parameters of the policy network and the commentator network are synchronously updated using the Independent Proximal Policy Optimization (IPPO) algorithm, and the control policy is output.

[0008] Preferably, the multimodal feature information includes: Text summary data: Key performance indicators (KPIs) used to describe the trajectory, including minimum obstacle avoidance distance, formation heading deviation, and relative distance between AUVs; Trajectory image data: used to reflect the geometry of multi-AUV formations and their relative positional relationship with obstacles, including trajectory smoothness, motion directionality, path efficiency, and obstacle avoidance performance.

[0009] Preferably, the large language model evaluator employs a dual scoring mechanism, and its calculation formula is as follows:

[0010] in, For KPI-based quantitative scores, including: (1) The distance maintenance index between the AUV and the environmental boundary is used to assess whether the AUV can travel stably along the central area of ​​the pipeline; (2) Formation heading consistency index, used to assess the heading deviation between the following AUV and the lead AUV; (3) Formation distance maintenance index, used to assess whether the relative distance between formation members is maintained near the preset target distance; This is a qualitative visual score based on trajectory images. The data comes from the current AUV trajectory images uploaded to the large language model. The large language model scores the trajectory from four dimensions based on the image, including... (1) Smoothness: The trajectory should avoid sharp turns or sharp jitters; (2) Directionality: The trajectory should generally remain forward, avoiding excessive backward movement or back-and-forth movement; (3) Path efficiency: The path should be relatively short and direct, without unnecessary detours; (4) Obstacle avoidance rationality: The trajectory should not intersect with obvious obstacles or boundaries.

[0011] in, and Preset weighting coefficients are used to balance the contributions of quantitative indicator evaluation and visual trajectory evaluation to the final score.

[0012] Preferably, the dynamic update mechanism for the expert trajectory pool includes: When the trajectory score output by the large language model exceeds the set sliding threshold, this trajectory is stored in a temporary demonstration buffer of size K. When the temporary demonstration buffer is full, all the trajectories within it are used to replace the original pool of suboptimal expert trajectories, thereby achieving iterative evolution of the expert demonstration level.

[0013] Preferably, the loss function formula of the discriminator network is:

[0014] in, This represents the discriminator network. Indicates the current state of the agent. Indicates the agent's current action. This indicates the probability that the discriminator determines the current state-action pair from the expert. Represents the trajectory of the intelligent agent. This represents the example trajectory in the expert trajectory pool. This represents the expectation operation on the distribution of the corresponding trajectory samples.

[0015] Preferably, the discriminator network generates a simulated reward signal. The calculation method is as follows:

[0016] in, This represents the discriminator's probability estimate of the current state-action pair from the expert demonstration; the reward signal is used to calculate the value V(s) and advantage function through the critic network, and to update the policy network parameters through the policy gradient method so that the trajectory generated by the agent gradually approximates the expert trajectory distribution; After each training cycle, the system inputs the trajectory generated by the agent into the large language model evaluator for scoring. When the score is higher than the average score of the current expert demonstration trajectory, the trajectory is stored in a temporary demonstration buffer. When the temporary demonstration buffer reaches a preset capacity, the demonstration data in the expert trajectory pool is replaced by the trajectory within it, thereby updating the discriminator training data and further guiding the policy network optimization.

[0017] Preferably, the update process of the independent proximal policy optimization algorithm includes: For each AUV, based on its own state and actions and the imitation reward signal generated by the discriminator, policy updates and value updates are performed independently. In each update step, the probability ratio of the old and new policies is calculated, and the policy update magnitude is limited by a shearing function. The shearing objective function of the policy network is:

[0018] in For probability ratios, The advantage function is used to characterize the degree of superiority or inferiority of the current action relative to the baseline state. This is a shearing factor used to limit the update magnitude between the old and new strategies; This represents the shearing function, used to restrict the probability ratio to an interval. Inside; This represents the expectation operation on the distribution of the corresponding trajectory samples.

[0019] Preferably, the design of the prompting project includes the following: Role setting: The large language model is preset as an underwater robot formation expert, enabling it to professionally evaluate the input trajectory based on the background of multi-AUV formation control and obstacle avoidance tasks; Evaluation criteria are set: a clear trajectory quality scoring standard is preset for the large language model, and the scoring standard includes at least formation stability score, path smoothness score and obstacle avoidance safety score; Output constraint setting: The large language model is required to output the evaluation results in a standardized JSON format, and the evaluation results shall include at least the scores of each sub-item and the final scalar score.

[0020] Preferably, the update logic of the expert trajectory pool follows the self-imitation principle, specifically including: The system sets a dynamic sliding threshold, which is adaptively determined based on the historical scoring statistics of the current expert trajectory pool. When the score of the trajectory generated in the current training round after evaluation by the large language model is higher than the sliding threshold, the trajectory is marked as a pseudo-expert demonstration and stored in the temporary demonstration buffer. When the temporary demonstration buffer meets the preset update conditions, the expert trajectory pool is updated using the pseudo-expert demonstration trajectories therein, so that the policy learning process can continuously absorb data samples that are better than the initial demonstration and gradually break through the performance limit of the initial expert demonstration.

[0021] Preferably, the application environment includes narrow pipes or variable-angle passages with random obstacles; the application tasks include multi-AUV formation maintenance, dynamic obstacle avoidance, and long-distance pipe inspection. Compared with the prior art, the present invention has the following beneficial effects: 1. This invention addresses the high dependence on human experts in traditional interactive imitation learning by introducing a large language model as a trajectory scorer, automating the scoring process and significantly reducing training costs. This invention employs a specific underlying mechanism design for the reinforcement learning loop, rather than allowing the large model to operate freely. Instead, it employs a clear dual scoring formula, embedding benchmark samples and combining quantitative indicators with qualitative image multimodal analysis.

[0022] 2. This invention employs a self-imitation learning mechanism, dynamically filtering the agent's generated trajectories through a large language model. This enables the system to learn from suboptimal initial data and gradually evolve policy performance that surpasses that of experts. To prevent policy collapse due to unstable large model scores, a "temporary trajectory pool" buffer mechanism is specifically designed at the system level. Batch replacement is only triggered when the trajectory score meets the standard and the buffer pool is full, directly suppressing the risk of misjudgment from an architectural perspective.

[0023] 3. MAGAISI designed for multi-AUV collaborative tasks The M-framework, a deeply coupled adversarial framework, directly alters the training data distribution of the discriminator network based on the selection results of the large model, thereby generating internal reward signals to guide policy updates. It exhibits higher formation-keeping accuracy and obstacle avoidance success rate in complex underwater environments (such as narrow culverts and dynamic obstacles).

[0024] 4. This invention effectively integrates human logical understanding of tasks (described through natural language) with the numerical optimization capabilities of deep reinforcement learning through prompting engineering, making the trained AUV behavior more consistent with actual physical laws and human aesthetics. Attached Figure Description

[0025] Figure 1 This is a flowchart of the multi-AUV formation and obstacle avoidance method based on large language model in-loop generation adversarial self-imitation learning provided by the present invention.

[0026] Figure 2 It consists of three AUV simulators deployed on the Gazebo platform and underwater environment scene diagrams.

[0027] Figure 3 This is a schematic diagram showing the observation status of the navigator AUV and two follower AUVs.

[0028] Figure 4 This is the cumulative reward curve from Task 1. It shows the cumulative rewards for the leader AUV(a) and follower AUV1(b). All methods were trained on the same suboptimal expert demonstration set for fair comparison.

[0029] Figure 5 These are actual trajectory graphs of the policy trained in Task 1 using MAGAISIL³M, evaluated across three tasks. (a) represents Task 1, (b) represents Task 2, and (c) represents Task 3.

[0030] Figure 6 This refers to the distance of the pilot AUV from the pipe wall or obstacle in three missions. (a) represents mission one, (b) represents mission two, and (c) represents mission three. The test used the final control strategies trained in mission one using both the MAGAISIL³M method based on suboptimal expert demonstration and the MAGAISIL method based on suboptimal expert demonstration.

[0031] Figure 7 This refers to the distance between the lead AUV and the follower AUV1 in three missions. (a) represents mission one, (b) represents mission two, and (c) represents mission three. The control strategy used is... Figure 6 Consistent.

[0032] Figure 8 This is a graph showing the Student's t-test results of the Fréchet distance between the trajectory generated by the AUV trained by MAGAISIL³M and MAGAISIL and the optimal expert demonstration. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. This embodiment aims to demonstrate how to automatically evaluate the training of multi-AUV formation and obstacle avoidance strategies using a large language model.

[0034] Example 1: This invention mainly consists of two core components that work together: a generative adversarial imitation component and a demonstration substitution component based on a large language model. The overall method flow is as follows: Figure 1 As shown.

[0035] This invention employs a classic leader-follower architecture to organize AUV formations. For example... Figure 2 As shown, the formation consists of one leader AUV and two follower AUVs. This configuration simplifies the problem while providing good scalability for larger AUV swarms. During the training and evaluation phases, each AUV makes independent decisions based on its own observations.

[0036] The Generative Adversarial Imitation (GAP) module is the core of strategy optimization, responsible for enabling AUV agents to continuously hone their skills through adversarial training. This includes the following steps: The system initially inputs a batch of excellent expert demonstration data.

[0037] In the distributed multi-AUV cooperative framework, each AUV agent independently trains a policy network and a critic network through IPPO. Simultaneously, a discriminator is trained to distinguish between state-action pairs generated by the agent and those demonstrated by the expert. The specific steps are as follows: Among them, the policy network (Actor, i.e., control policy) The agent i is responsible for making decisions; that is, during the training process at time step t, the agent i receives the currently perceived local environmental state. And sample an action based on its current policy. :

[0038] Where i represents the AUV agent number and t represents the time step. This represents the state of agent i at time step t. This represents the action sampled by agent i at time step t. The parameter is The policy network, This represents the policy network parameters. After performing this action, the agent transitions to the next state, and this process repeats until an episode ends. The system stores this series of consecutive state-action pairs (…). , Accumulated records form the complete motion trajectory of the intelligent agent. Its goal is to maximize expected long-term returns, bringing the trajectory closer to expert levels.

[0039] Critic networks (i.e., value networks) Essentially, it is a value function estimator, responsible for evaluating the quality of a state by calculating its value. It is used to evaluate the expected return of the current strategy. During training, it provides the Actor with an advantage function signal, reduces the variance of gradient estimation, and guides the Actor to update in the direction of higher returns.

[0040] Trajectory generated by the agent Randomly selected from Group state-action, and from expert trajectory pool Extract the same number of expert state-action pairs and put them into the discriminator. Training is then performed. The discriminator's task is to distinguish whether an action originates from an agent or an expert. Its loss function is formulated as follows:

[0041] in, This represents the discriminator network. Indicates the current state of the agent. Indicates the agent's current action. This indicates the probability that the discriminator determines the current state-action pair from the expert. Represents the trajectory of the intelligent agent. This represents the example trajectory in the expert trajectory pool. This represents the expectation operation on the distribution of the corresponding trajectory samples.

[0042] After the discriminator is trained, a reward value will be provided to the agent. The more the agent's behavior resembles that of an expert, the higher the reward, and the harder it is for the discriminator to distinguish whether the action comes from the agent or the expert. The reward function formula is:

[0043] This reward signal is then used to update the policy network and commentator network of agent i.

[0044] The demonstration replacement module based on a large language model uses a prompt-engineered GPT-4 model to act as an expert evaluator. It accesses the system via an API. The expert evaluator's score is weighted by a "KPI-based quantitative score (score1, 70%)" and an "image-based qualitative score (score2, 30%)," assigned to each trajectory on a scale of 1 to 100 according to the following task requirements. score.

[0045] KPI-based quantitative scoring (score1) is based on the following task requirements: 1. The average distance of all AUVs from the pipe wall should be as close as possible to the optimal value of 17.3 meters; 2. The heading deviation between the lead AUV and the follower AUV should be close to 0; 3. The relative distance between AUVs should be kept stable at the target value of about 18 meters.

[0046] The closer a trajectory’s KPIs are to these requirements, the higher the score the large language model is likely to give, thus increasing the chance that the trajectory will be selected and added to a temporary demonstration buffer for policy optimization.

[0047] The image-based qualitative score, score2, is derived from the current AUV trajectory image uploaded to the large language model. The large language model scores the trajectory based on the image from four dimensions: 1. Smoothness: The trajectory should avoid sharp turns or jerks; 2. Directionality: The trajectory should generally remain forward, avoiding excessive backward movement or back-and-forth movement; 3. Path efficiency: The path should be relatively short and direct, without unnecessary detours; 4. Obstacle avoidance rationality: The trajectory should not intersect with obvious obstacles or boundaries.

[0048] Among them, score1 provides a robust quantitative evaluation based on preset metrics, while score2 allows large language models to evaluate qualitative aspects (such as smoothness and naturalness) from trajectory images, features that are difficult to capture using simple KPIs. This dual scoring mechanism constructs a more comprehensive evaluation system. The final score is calculated using empirical weights as follows: . The specific steps of the evaluation process are as follows: At the end of each training cycle, the system receives the trajectory generated by agent i. The KPIs of the trajectory are extracted and submitted as text prompts to the large language model for analysis.

[0049] The large language model, based on the above task requirements, defines each trajectory on a scale from 1 to 100. Rate it.

[0050] If the score is higher than the current expert demonstration The average score is then considered a trajectory that "performs better than the expert's" and is stored in a temporary demonstration buffer of size K=10. Otherwise discard.

[0051] Only when the buffer is full will all stored trajectories immediately replace the current expert demonstration. Subsequently, The data is cleared so that new candidate trajectories can be collected in the next training phase. This buffering mechanism can mitigate the impact of any single anomalous evaluation, ensuring that improvements on expert demonstration data are based on consistent high performance, rather than sporadic, potentially erroneous scores. Example 2: To verify the effectiveness of the present invention, the technical solution and technical effects of the present invention will be further explained below in conjunction with specific experiments.

[0052] This embodiment uses the Gazebo physics simulation engine based on the Ubuntu system. Gazebo is an open-source, physics-based 3D robot simulator that supports multibody dynamics, sensor modeling, and real-time visualization. It is widely used for algorithm verification and hardware-in-the-loop testing of marine, land, and aerial robots. Its underwater extension module—the Unmanned Underwater Vehicle Simulator—integrates complete hydrodynamic plugins, sonar and Doppler sensor models, and a Robot Operating System (ROS) interface into Gazebo, enabling high-fidelity reproduction of underwater communication delays, flow field disturbances, and collision responses.

[0053] The AUV model used in this embodiment is the SailFish 210 model, which possesses realistic six-DOF AUV parameters, a symmetrical twin-propeller propulsion system, a forward-looking sonar array, and a pressure-resistant hull with a rated depth of 300 meters. It accurately simulates navigation, hovering, and formation coordination behavior in a virtual environment, providing reliable and repeatable experimental evidence for evaluating the proposed multi-AUV formation and obstacle avoidance algorithm. Figure 2 The study showcased three AUV simulators deployed on the Gazebo platform and their simulated underwater environments, which were used in our research.

[0054] To comprehensively and rigorously evaluate the performance of the proposed method, this embodiment constructs three simulation scenarios with increasing difficulty, focusing on the formation control and dynamic obstacle avoidance capabilities of multi-AUV systems. Each task is designed to test the robustness and adaptability of the algorithm under specific challenges. In these tasks, the observation states of the leader AUV and the two follower AUVs are as follows: Figure 3 As shown.

[0055] Task 1: Baseline Formation Navigation and Boundary Preservation: This task serves as a baseline scenario to verify the algorithm's basic formation preservation and path tracking capabilities. Specifically, the lead AUV must navigate within a straight, rectangular cross-section pipe with a width of 30 meters. Two follower AUVs must maintain a straight formation behind the lead AUV and keep a fixed distance from it. Simultaneously, all AUVs must maintain a safe lateral distance from the pipe wall. In this task, the starting coordinates of the lead AUV, follower AUV1, and follower AUV2 are set to (18, 15), (9, 15), and (1, 15), respectively, with a common target point at (240, 15).

[0056] Task 2: Reactive Obstacle Avoidance in Dense Environments: To test the algorithm's rapid response capability in complex and unknown environments, this task introduces dense static obstacles based on Task 1. Multiple spherical obstacles with a radius of 5 meters are strategically placed at different locations within the pipe (such as the center of the channel, near curves, and along the wall) to simulate a more complex and dangerous underwater obstacle environment.

[0057] Task 3: High-Mobility Formation Control on Complex Paths: This task aims to evaluate the stability and control accuracy of a formation during extensive maneuvers. The pipeline geometry was modified, adding "zigzag" sharp turns and extending the total length to 300 meters. The AUV formation was required to maintain formation integrity throughout high-speed turns.

[0058] In all tasks, the current training cycle will be terminated and a new cycle will begin if any of the following conditions are met: 1. Any AUV must be less than 1 meter away from an obstacle (pipe wall): or or . in, Indicates the distance from the pilot AUV to the nearest obstacle. This indicates the distance from AUV1 to the nearest obstacle. This indicates the distance to the nearest obstacle while following AUV2.

[0059] 2. The relative distance between following AUVs exceeds [1, 33] meters: or or or . in, This indicates the distance between the lead AUV and the following AUV1. This indicates the distance between following AUV1 and following AUV2. For example, The calculation formula is as follows:

[0060] in, Indicates the coordinates of the Navigator AUV. This indicates the coordinates following AUV1.

[0061] 3. Any deviation in heading from the AUV's course exceeding... radian: or . in, This indicates the heading deviation following AUV1. This indicates the heading deviation while following AUV2. For example, The calculation formula is:

[0062] Among them, variables This indicates the current heading of follower AUV1.

[0063] In the above three tasks, the AUV state space and motion space are set as follows: AUV state space This includes its relative distance in spherical coordinates, yaw angle, and distances to obstacles in the six nearest directions detected by sonar. The sonar data comes from the front-mounted sonar sensor on each AUV. The sensor's detection angle is... It is divided into 6 independent sectors (e.g. Figure 3 (As shown in the shaded area) to provide orientation awareness. The sensor is configured with 600 beams, evenly distributed across these sectors, with a maximum effective detection range of 33 meters. The shortest distance detected across all sectors is used as the critical obstacle distance for decision-making. The state of each AUV is represented as a vector of its local observations. The shortest obstacle distances detected by the navigator and follower AUVs are respectively... , and Indicates the status of the Navigator AUV (AUV). It consists of the nearest obstacle distances detected by only 6 sonar sectors:

[0064] in, This indicates the distance to the nearest obstacle detected by sonar in different sectors for the navigator AUV. Follower AUV status. In addition to the readings from each of the six sonar sectors, the relative attitude information of its reference subject (leader or preceding follower) is also included:

[0065] in, This represents the Euclidean distance between AUVs.

[0066] The AUV's maneuver space is discretized into five basic maneuvers. These maneuvers include a small left turn, a large left turn, a small right turn, a large right turn, and straight-line movement, achieved by adjusting the thruster speed and rudder angle. In the 2D simulation plane of this embodiment, only the upper and lower rudders are used to control horizontal movement; the left and right rudders used for buoyancy control are fixed at zero. The thruster speed of all AUVs is kept constant at 300 r / s to ensure a consistent baseline speed.

[0067] Since the discriminator reward obtained based on expert demonstrations essentially originates from distribution matching, this embodiment designs a set of task-constraint-based reward functions for all AUVs, specifically for evaluating the performance of the learned policy. It is important to emphasize that these reward functions are used only during the testing phase and are not involved in the training process. This ensures that different algorithms receive fair and unbiased evaluations.

[0068] The reward function for the Navigator AUV is:

[0069] In the formula, This represents the shortest distance between the pilot AUV and the obstacle. 17.3 m is a safe distance derived from the detection angle range of the sonar sensor and the distance between the two side walls of the pipeline; this distance allows the AUV to remain centered within the pipeline. It can be seen that the more the pilot AUV deviates from the center of the pipeline, the lower its reward value; conversely, the closer it is to the center, the higher its reward value.

[0070] The reward function for follower AUV is:

[0071] In the formula, This represents the follower's reward for tracking the leader. This represents its obstacle avoidance reward. Since the follower AUV1 needs to simultaneously perform formation tracking and obstacle avoidance tasks, the two rewards are assigned the same weight (0.5) to achieve balance.

[0072] in, Used to evaluate its formation maintenance capability with respect to a reference AUV, specifically defined as:

[0073] In the formula, To maintain the formation's rewards, For heading deviation, This is the actual distance relative to a reference AUV. This item also constrains heading deviation. and relative distance The degree of deviation from the target value of 18 m.

[0074] The second part, similar to the rewards for the Navigator AUV, is designed to encourage followers to stay on the center line of the channel at all times, and is specifically defined as:

[0075] In the formula, To maintain the reward for the center line, This represents the follower's actual distance from the center line. The reward uses 17.3 m as the optimal center position and incentivizes the follower to stay on the center line of the passage by penalizing deviation.

[0076] While designing such a comprehensive reward function is feasible, the process is extremely time-consuming and requires extensive domain expertise and iterative parameter tuning. In contrast, using suboptimal expert demonstrations for policy guidance is a more efficient way to introduce knowledge.

[0077] In the experiments described in this invention, both the proposed method and the baseline method employ the Actor-Critic framework. Both the policy network (Actor) and the value network (Critic) are implemented using fully connected networks (FCNs). The specific network architecture is: FCN(Actor) = [64, 128, 64, 5, 5], FCN(Critic) = [64, 128, 64, 1]. Tanh is used as the activation function for the hidden layers. Softmax is used for the output layer of the Actor, while no activation function is set for the output layer of the Critic.

[0078] During training, the mini-batch size is set to N = 256. In each update, 256 state-action pairs are randomly sampled from the trajectories generated by each AUV and their corresponding expert demonstrations to update the discriminator. To maintain learning stability, the discriminator is updated 3 times per round, while the generator (i.e., the policy network) for each AUV is updated 9 times to mitigate policy fluctuations caused by discriminator changes. For the IPPO algorithm, the discount factor... Set it to 0.99, Set to 1.0, clipping factor Set to 0.09. The maximum capacity of the temporary demonstration buffer is set to 10 tracks.

[0079] To verify the MAGAISI proposed in this invention The effectiveness of the M-method was evaluated by comparing it with two benchmark methods, MAGAIL and MAGAISIL. Experiments were conducted in Task 1 using suboptimal expert demonstration data. To test generalization ability, the trained policies of the three AUV agents were deployed to the more challenging environments of Tasks 2 and 3.

[0080] Experimental results: Figure 4 Figures (a) and (b) respectively demonstrate the cumulative rewards of the leader and follower AUVs, comparing them to MAGAIL, MAGAISIL, and the proposed MAGAISIL³M. To ensure fairness in the comparison, all methods were trained using the same suboptimal demonstration data. The performance of the original optimal and suboptimal expert demonstrations is also plotted as a theoretical benchmark.

[0081] like Figure 4As shown in (a), MAGAISIL and MAGAISIL³M consistently outperform the baseline method MAGAIL throughout the training process. Notably, MAGAISIL³M exhibits a significantly faster learning rate, converging to the optimal reward region within approximately 300 training epochs. This indicates that the trajectory filtering mechanism driven by the large language model significantly accelerates policy optimization. In contrast, the MAGAIL method struggles to surpass suboptimal performance, highlighting the limitations of learning directly from low-quality examples.

[0082] Figure 4 The learning curve of the follower AUV1 shown in Figure (b) also exhibits a similar trend. Compared to MAGAIL, both MAGAISIL and MAGAISIL³M achieve faster convergence and better final performance. Among them, MAGAISIL³M once again demonstrates the fastest convergence.

[0083] This is thanks to the ability of large language models to evaluate trajectory quality in a consistent and automated manner. While MAGAISIL, a "human-in-the-loop" approach, improves upon MAGAIL, its performance is constrained by the inherent limitations of human evaluation: human trainers' judgments can be influenced by cognitive biases, fatigue, and incomplete memories of past trajectories, potentially leading to inconsistent or delayed replacement decisions. In contrast, MAGAISIL³M provides consistent, objective, and tireless evaluation signals. The ability of large language models to process both quantitative metrics and qualitative visual data without bias, resulting in more reliable and efficient selection of high-quality trajectories, is a key driver of its accelerated convergence.

[0084] In summary, MAGAISIL³M effectively overcomes the performance bottleneck of learning from low-quality data. By progressively constructing high-quality training datasets through its large language model-assisted demonstration replacement mechanism, the framework validates its significant advantages in sample efficiency and policy convergence.

[0085] To evaluate the effectiveness of the final control strategy in Mission 1, this embodiment conducted 100 trials, comparing the performance of the navigator and follower AUVs trained using the MAGAISIL³M method (based on suboptimal expert demonstration training) and the MAGAISIL method (also based on suboptimal demonstration training) proposed in this embodiment. The evaluation was conducted across multiple dimensions, including trajectory tracking, distance to surrounding walls or obstacles, AUV spacing (between the navigator and follower AUV1), and the heading deviation of follower AUV1. These results are presented in... Figure 5 (a) Figure 6 (a) and Figure 7 The visualization is shown in (a).

[0086] like Figure 6As shown in (a), the MAGAISIL³M maintained a stable and safe distance between the navigator AUV and the wall during operation, performing comparably to the MAGAISIL, and exhibiting more consistent peak characteristics when performing obstacle avoidance maneuvers. This demonstrates that the MAGAISIL³M can effectively learn key turning behaviors from suboptimal demonstrations while still maintaining a safety margin, highlighting its superior stability and responsiveness when following the navigator and simultaneously avoiding collisions.

[0087] Regarding formation alignment Figure 7 (a) shows that MAGAISIL³M maintains a larger and more stable agent spacing throughout the round. This increased spacing reflects conservative and safety-oriented behavior guided by the large language model evaluation during trajectory replacement, which helps with obstacle avoidance and improves formation integrity.

[0088] To demonstrate the naturalness and smoothness of the trajectory, this embodiment selects an optimal expert demonstration that excels in these aspects, and equates the naturalness and smoothness of the generated trajectory with its similarity to this optimal expert demonstration. This embodiment chooses to use the Fréchet distance to quantify this metric: . Where A and B are two parametric curves. : [0,1] [0,1] is any non-decreasing reparameterized function. Represents Euclidean distance. Smaller The value indicates a higher similarity, meaning higher naturalness and smoothness.

[0089] Figure 8 The Student's t-test results for the Fréchet distance between the trajectories generated by AUVs trained with MAGAISIL³M and MAGAISIL and the best expert demonstration are presented in Task 1. In this context, a lower Fréchet distance implies higher trajectory similarity, meaning the AUV's trajectory is closer to the natural, smooth trajectory of the expert demonstration. The trajectories generated by MAGAISIL³M, evaluated by the large language model, are significantly ( The trajectories generated by MAGAISIL³M are smoother and more natural than those generated by MAGAISIL after human evaluation. It can be observed that the trajectories generated by MAGAISIL³M are smoother than those generated by MAGAISIL. Overall, these results validate that MAGAISIL³M can effectively learn stable, safe, and coordinated behavior in multi-AUV formation tasks, even when trained using suboptimal examples, outperforming benchmark methods in obstacle avoidance consistency and formation alignment stability.

[0090] To evaluate the adaptability of the MAGAISIL³M method in handling complex and variable tasks, the final control strategies of the navigator AUV and follower AUV1 and AUV2, trained using both MAGAISIL³M and MAGAISIL methods, were applied to Task 2 (adding additional obstacles) and Task 3 (changing the wall angle and extending the pipe length). These tasks were set up logically consistent with Task 1, aiming to test the performance of the two methods when facing different challenges.

[0091] like Figure 5 (b)(c) Figure 6 (b)(c) and Figure 7 As shown in (b) and (c), this embodiment compares the performance of the two methods in terms of trajectory tracking, distance to obstacles, and the spacing between the navigator and follower AUV1. The analysis shows that the control strategies trained by both methods can adapt well to complex tasks involving more obstacles or changes in wall angles.

[0092] It is worth noting that the MAGAISIL³M method demonstrates its advantages in several aspects. Particularly at pipe bends, AUVs trained with MAGAISIL³M slightly outperform those trained with MAGAISIL (e.g., ...). Figure 5 (As shown in (b) and (c)). This demonstrates that the MAGAISIL³M method can adjust its strategy more quickly to adapt to changes in dynamic environments. In Task 2, both methods performed similarly when faced with additional obstacles. However, in Task 3, where the environment changed more drastically, the MAGAISIL³M method exhibited more stable performance.

[0093] Specifically, in Mission 3, the follower AUV1 trained by MAGAISIL³M performed more consistently in maintaining a specific distance from the navigator (e.g., Figure 7 As shown in (c), this is crucial for maintaining formation consistency and navigational safety, as it helps reduce formation disorder caused by sudden environmental changes.

[0094] In summary, although the two methods performed similarly in Task 2, the MAGAISIL³M method demonstrated a more significant advantage in stability and formation maintenance in Task 3. These results indicate that the MAGAISIL³M method is more suitable for applications with significant environmental changes, as it can provide a more stable and reliable control strategy.

[0095] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0096] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for generating formation and obstacle avoidance strategies for multiple autonomous underwater vehicles, characterized in that, Includes the following steps: S1, initialize the multi-AUV environment and the policy network, critic network, discriminator network and expert trajectory pool of each AUV agent, as well as a temporary demonstration buffer for storing candidate high-quality trajectories; S2, In the training round, each AUV agent generates and executes control actions based on the current perceived local environment state using the policy network, forming the motion trajectory data for the current round, and extracting the multimodal feature information of the trajectory; S3, Construct a large language model evaluator, which transforms the multimodal feature information into text descriptions and key indicator data that the large language model can parse through prompting engineering, and the large language model scores the trajectory quality. S4. The expert trajectory pool is dynamically updated based on the scoring results. First, the current trajectory is filtered, and the high-scoring trajectory is stored in a temporary demonstration buffer. When the preset conditions are met, the trajectory in the temporary demonstration buffer is used to update the expert trajectory pool. S5, sample state-action pairs from the updated expert trajectory pool and the trajectory generated by the current AUV respectively, train the discriminator network to distinguish between expert trajectories and agent trajectories, and generate a mimicry reward signal; S6, combining the simulated reward signal, the parameters of the policy network and the critic network are synchronously updated through the independent proximal policy optimization algorithm, and the control policy is output.

2. The method for generating multi-autonomous underwater vehicle formation and obstacle avoidance strategies as described in claim 1, characterized in that: The multimodal feature information includes: Text summary data: Key performance indicators used to describe the trajectory, including minimum obstacle avoidance distance, formation heading deviation, and relative distance between AUVs; Trajectory image data: used to reflect the geometry of multi-AUV formations and their relative positional relationship with obstacles, including trajectory smoothness, motion directionality, path efficiency, and obstacle avoidance performance.

3. The method for generating multi-autonomous underwater vehicle formation and obstacle avoidance strategies as described in claim 1, characterized in that: The large language model evaluator described above employs a dual scoring mechanism, and its calculation formula is as follows: in, For KPI-based quantitative scores, including: (1) The distance maintenance index between the AUV and the environmental boundary is used to assess whether the AUV can travel stably along the central area of ​​the pipeline; (2) Formation heading consistency index, used to assess the heading deviation between the following AUV and the lead AUV; (3) Formation distance maintenance index, used to assess whether the relative distance between formation members is maintained near the preset target distance; This is a qualitative visual score based on trajectory images. The data comes from the current AUV trajectory images uploaded to the large language model. The large language model scores the trajectory based on the image from four dimensions, including: (1) Smoothness: The trajectory should avoid sharp turns or sharp jitters; (2) Directionality: The trajectory should generally remain forward, avoiding excessive backward movement or back-and-forth movement; (3) Path efficiency: The path should be relatively short and direct, without unnecessary detours; (4) Obstacle avoidance rationality: The trajectory should not intersect with obvious obstacles or boundaries; in, and Preset weighting coefficients are used to balance the contributions of quantitative indicator evaluation and visual trajectory evaluation to the final score.

4. The method for generating multi-autonomous underwater vehicle formation and obstacle avoidance strategies as described in claim 1, characterized in that: The expert trajectory pool dynamic update mechanism includes: When the trajectory score output by the large language model exceeds the set sliding threshold, this trajectory is stored in a temporary demonstration buffer of size K. When the temporary demonstration buffer is full, all the trajectories within it are used to replace the original pool of suboptimal expert trajectories, thereby achieving iterative evolution of the expert demonstration level.

5. The method for generating multi-autonomous underwater vehicle formation and obstacle avoidance strategies as described in claim 1, characterized in that: The loss function formula for the discriminator network is as follows: in, This represents the discriminator network. Indicates the current state of the agent. Indicates the agent's current action. This indicates the probability that the discriminator determines the current state-action pair from the expert. Represents the trajectory of the intelligent agent. This represents the example trajectory in the expert trajectory pool. This represents the expectation operation on the distribution of the corresponding trajectory samples.

6. The method for generating multi-autonomous underwater vehicle formation and obstacle avoidance strategies as described in claim 1, characterized in that: The discriminator network generates a mimicry reward signal. The calculation method is as follows: in, This represents the discriminator's probability estimate of the current state-action pair from the expert demonstration; the reward signal is used to calculate the value V(s) and advantage function through the critic network, and to update the policy network parameters through the policy gradient method so that the trajectory generated by the agent gradually approximates the expert trajectory distribution; After each training cycle, the system inputs the trajectory generated by the agent into the large language model evaluator for scoring. When the score is higher than the average score of the current expert demonstration trajectory, the trajectory is stored in a temporary demonstration buffer. When the temporary demonstration buffer reaches a preset capacity, the demonstration data in the expert trajectory pool is replaced by the trajectory within it, thereby updating the discriminator training data.

7. The method for generating multi-autonomous underwater vehicle formation and obstacle avoidance strategies as described in claim 1, characterized in that: The update process of the independent proximal policy optimization algorithm includes: For each AUV, based on its own state and actions and the imitation reward signal generated by the discriminator, policy updates and value updates are performed independently. In each update step, the probability ratio of the old and new policies is calculated, and the policy update magnitude is limited by a shearing function. The shearing objective function of the policy network is: in For probability ratios, The advantage function is used to characterize the degree of superiority or inferiority of the current action relative to the baseline state. This is a shearing factor used to limit the update magnitude between the old and new strategies; This represents the shearing function, used to restrict the probability ratio to an interval. Inside; This represents the expectation operation on the distribution of the corresponding trajectory samples.

8. The method for generating multi-autonomous underwater vehicle formation and obstacle avoidance strategies as described in claim 1, characterized in that: The design of the prompting project includes the following: Role setting: The large language model is preset as an underwater robot formation expert, enabling it to professionally evaluate the input trajectory based on the background of multi-AUV formation control and obstacle avoidance tasks; Evaluation criteria are set: a clear trajectory quality scoring standard is preset for the large language model, and the scoring standard includes at least formation stability score, path smoothness score and obstacle avoidance safety score; Output constraint setting: The large language model is required to output the evaluation results in a standardized JSON format, and the evaluation results shall include at least the scores of each sub-item and the final scalar score.

9. The method for generating multi-autonomous underwater vehicle formation and obstacle avoidance strategies as described in claim 1, characterized in that: The update logic of the expert trajectory pool follows the self-imitation principle, specifically including: The system sets a dynamic sliding threshold, which is adaptively determined based on the historical scoring statistics of the current expert trajectory pool. When the score of the trajectory generated in the current training round after evaluation by the large language model is higher than the sliding threshold, the trajectory is marked as a pseudo-expert demonstration and stored in the temporary demonstration buffer. When the temporary demonstration buffer meets the preset update conditions, the expert trajectory pool is updated using the pseudo-expert demonstration trajectories therein.

10. A method for generating multi-autonomous underwater vehicle formation and obstacle avoidance strategies as described in any one of claims 1 to 9, characterized in that: Application environments include narrow pipes or variable-angle passages with random obstacles; application tasks include multi-AUV formation maintenance, dynamic obstacle avoidance, and long-distance pipe inspection.

Images