Deep reinforcement learning formation control method for underwater vehicle

Abstract

The invention discloses an underwater vehicle deep reinforcement learning formation control method based on an imitation learning compensator, and an underwater vehicle reinforcement learning model comprises a first decision network and N second decision networks, input of a first decision network is a state of a first underwater vehicle, output of the state of the first underwater vehicle is an action of the first underwater vehicle, input of a second decision network is a state of a second underwater vehicle corresponding thereto, and output of the action of the second underwater vehicle is an action of the second underwater vehicle when the second underwater vehicle cannot acquire the action of the first underwater vehicle According to the underwater vehicle deep reinforcement learning formation control method, state perception is provided for the second underwater vehicle, so that the second underwater vehicle accurately prejudges the navigation intention of the first underwater vehicle, and a better formation is kept.

Description

Technical Field

[0001] This invention belongs to the field of underwater vehicle control technology, specifically relating to a deep reinforcement learning formation control method for underwater vehicles. Background Technology

[0002] As a new type of underwater robot, underwater vehicles have received extensive research attention both domestically and internationally in recent years. Equipped with sensors, navigation systems, and power units, underwater vehicles can autonomously complete tasks based on pre-programmed instructions or real-time data. Therefore, underwater vehicles are characterized by high autonomy, high safety, high efficiency, and low cost.

[0003] In recent years, with the continuous deepening of ocean exploration, a single underwater vehicle can no longer meet the needs of exploration, and forming a fleet of multiple underwater vehicles has become an important issue. Deploying multiple underwater vehicles simultaneously can expand the coverage area in the water at the same time, facilitating three-dimensional and three-dimensional exploration.

[0004] However, the attitude of current underwater vehicles is easily affected by ocean currents and waves, and their navigation trajectories differ significantly from the preset trajectories. The method of pre-setting trajectories for each underwater vehicle to maintain formation is difficult to overcome this deviation. This can easily lead to unsatisfactory formations or even breakup of the formation due to different veer directions of the underwater vehicles.

[0005] Furthermore, when using a master-slave topology formation for multiple underwater vehicles, the state perception between the underwater vehicles is easily affected by factors such as communication delays and interruptions. This makes it difficult to ensure that the control algorithm can obtain the motion state information of the first underwater vehicle in a timely manner, which makes it difficult to maintain a stable formation.

[0006] In summary, there is an urgent need for a deep reinforcement learning-based formation control method for underwater vehicles to address the problems of unsatisfactory formations and spatiotemporal asynchrony that arise when underwater vehicles are controlled by pre-set trajectories. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a reinforcement learning model for underwater vehicles.

[0008] Another objective of this invention is to provide a training method for a reinforcement learning model of an underwater vehicle.

[0009] The present invention is achieved through the following technical solutions.

[0010] An underwater vehicle reinforcement learning model includes: a first decision network and N second decision networks, where N≥1. Each first decision network corresponds to a first underwater vehicle, and each second decision network corresponds to a second underwater vehicle. The first underwater vehicle navigates along a route, and the second underwater vehicle follows the first underwater vehicle while maintaining a fixed distance and angle. The first and second underwater vehicles each make a decision every Δt time interval.

[0011] The input to the first decision network is the state of the first underwater vehicle at the t-th decision. The output of the first decision network is the action of the first underwater vehicle.

[0012] The input to the second decision network is the state of the corresponding second underwater vehicle at the t-th decision. and actions k is a constraint coefficient, and the output of the second decision network is the action of the second underwater vehicle.

[0013] The state of the first underwater vehicle at decision t (x g ,y g ,z g (x) represents the coordinates of the current waypoint of the first underwater vehicle in the world coordinate system. l ,y l , z l Let θ be the coordinates of the t-th decision point of the first underwater vehicle in the world coordinate system. g Let θ be the expected yaw angle when the first underwater vehicle reaches the current target waypoint. l u is the actual yaw angle when the first underwater vehicle reaches the current target waypoint. l v l w l The first underwater vehicle is located at coordinate point (x) l y l , z l At that time, in its body coordinate system, along x... l y l z l velocity in the direction, ω l The first underwater vehicle is located at coordinate point (x l ,y l ,z l angular velocity at time x l y k z k The directions are, in order, the x-axis, y-axis, and z-axis directions in the body coordinate system of the first underwater vehicle.

[0014] action In the body coordinate system of the first underwater vehicle, the first underwater vehicle moves along the x-axis. k y k z k Acceleration in the direction of The angular acceleration of the first underwater vehicle;

[0015] The state of the second underwater vehicle at decision t (x f ,y f ,z f Let θ be the coordinates of the t-th decision point of the second underwater vehicle in the world coordinate system, and let a, b, and c be the offsets of the initial position of the second underwater vehicle relative to the initial position of the first underwater vehicle in the x, y, and z directions, respectively. f Let u be the actual yaw angle of the second underwater vehicle's t-th decision. f v f w f The following are the coordinates of the second underwater vehicle at its t-th decision along x in the body coordinate system of the second underwater vehicle. f′ y f′ z f′ velocity in the direction, ω f Let x be the angular velocity of the second underwater vehicle. f′ y f′ z f′ The directions are, in order, the x-axis, y-axis, and z-axis directions in the body coordinate system of the second underwater vehicle;

[0016] The second underwater vehicle's actions The second underwater vehicle is positioned in the body coordinate system of the second underwater vehicle at x f′ y f′ z f′ Acceleration in the direction of This represents the angular acceleration of the second underwater vehicle.

[0017] In the above technical solution, the first decision network is constructed using a neural network, and the second decision network is constructed using a neural network.

[0018] The above technical solution also includes: a simulation learning compensator corresponding to each second underwater vehicle, wherein the input of the simulation learning compensator is the state. Imitate the output action of the learning compensator As an action The imitation learning compensator is constructed using a neural network.

[0019] The training method for the underwater vehicle reinforcement learning model described above involves performing a voyage once every 2T training cycles using the updated first decision network and the updated second decision network. If the first underwater vehicle and N second underwater vehicles can maintain formation and reach the mission endpoint in at least 5 consecutive voyages, then the training is stopped.

[0020] Each first sample is: state action state and reward value state Perform actions for the first underwater vehicle The reward value of the state after the (t+1)th decision Let be the reward value for the first underwater vehicle's decision t, calculated using a reward function:

[0021]

[0022] In the formula, θ′ l The state corresponding to the first underwater vehicle at decision t+1. θ l e α For the first underwater vehicle in state The angle between the vector pointing from the current location's coordinates to the current target waypoint and its own velocity vector, k 11 and k 12 All are coefficients;

[0023] Each second sample is: state action state Reward Value action and actions state Performing actions for the second underwater vehicle The state and action after the (t+1)th decision To input state into the first decision network The action obtained The product of k and the reward value Let be the reward value for the second underwater vehicle's decision t, calculated using a reward function:

[0024]

[0025] In the formula, k 21 and k 22 All are coefficients, i∈[1,N], d i For the i-th second underwater vehicle in state The distance between the current location's coordinates and its desired location's coordinates; e βi For the i-th second underwater vehicle in state The angle between the vector pointing from the current coordinate point to the desired position and its own velocity vector;

[0026] The training methods for reinforcement learning models of underwater vehicles include S1 and S2:

[0027] S1. Randomly select M first samples from one voyage of the first underwater vehicle. Set up evaluation networks Q1, Q2, Q3, Q4, and decision network P1. Update the first decision network, evaluation network Q1, and evaluation network Q2 sequentially with the M first samples. The method for updating the first decision network, evaluation network Q1, and evaluation network Q2 sequentially for each first sample is as follows:

[0028] The state in the first sample The input is fed into the first decision network to obtain the actions of the first underwater vehicle. As an action The state in the first sample and actions The value is obtained by inputting the data into the evaluation network Q1. The state in the first sample and actions The value is obtained by inputting the data into the evaluation network Q2. The state in the first sample The input is fed into decision network P1 to obtain the actions of the first underwater vehicle. As an action

[0029] State and actions The value is estimated by inputting it into the evaluation network Q3. State and actions The value is estimated by inputting it into the evaluation network Q4. Selection of value estimation and valuation The smaller value in the middle is used as the value estimate Based on value estimation and the reward value in the sample The value estimate was calculated. Based on value estimation Value estimation Value estimation state and actions Calculate the loss function and update the first decision network, evaluation network Q1, and evaluation network Q2 of the first underwater vehicle;

[0030] S2, randomly select M second samples from each second underwater vehicle during a single voyage. Each second underwater vehicle is configured with an evaluation network Q5, an evaluation network Q6, an evaluation network Q7, an evaluation network Q8, and a decision network P2. The "N second decision networks and their corresponding evaluation networks Q5 and Q6" are updated multiple times. The update method for each second decision network and its corresponding evaluation networks Q5 and Q6 is as follows:

[0031] The state of the second underwater vehicle in a single voyage during the second decision network's journey in a second sample. and actions The input is fed into the second decision network to obtain the action. As an action The states from one second sample of each of the N second underwater vehicles and actions The value estimate is obtained by inputting the data into the evaluation network Q5 corresponding to the second decision network. The states from one second sample of each of the N second underwater vehicles and actions The value estimate is obtained by inputting the data into the evaluation network Q6 corresponding to the second decision network. The state in the second sample corresponding to the second decision network and actions The input is fed into decision network P2, corresponding to the second decision network, to obtain the actions of the second underwater vehicle. As an action

[0032] The states from one second sample of each of the N second underwater vehicles The individual actions of N second underwater vehicles The inputs are combined with the evaluation network Q7 corresponding to the second decision network to obtain the output value estimate. The states from one second sample of each of the N second underwater vehicles and the actions of all second underwater vehicles The values ​​are then input together into the evaluation network Q8 corresponding to the second decision network to obtain the value estimate. Selection of value estimation and valuation The smaller value in the middle is used as the value estimate

[0033] Based on value estimation and the reward value in the sample Calculate the target value estimate Based on value estimation Value estimation Target value estimation The state of the second sample of N second underwater vehicles And the actions of N second underwater vehicles Calculate the loss function and update the second decision network, evaluation network Q5, and evaluation network Q6 for each second underwater vehicle.

[0034] In the above technical solution, the first decision network, evaluation network Q1, evaluation network Q2, second decision network, evaluation network Q5, and evaluation network Q6 are updated according to the MATD3 method: the first decision network is updated once for every n updates to evaluation network Q1 and evaluation network Q2; the second decision network is updated once for every n updates to evaluation network Q5 and evaluation network Q6.

[0035] In the above technical solution, the method for obtaining "one first sample and one second sample corresponding to each second underwater vehicle" each time is as follows:

[0036] ①The status of the first underwater vehicle The input is fed into the first decision network to obtain the action. Actions will be communicated Transmit the status of each second underwater vehicle to each second underwater vehicle. and actions The inputs are respectively fed into the second decision network corresponding to the second underwater vehicle to obtain the actions of the second underwater vehicle.

[0037] ② Enable the first underwater vehicle to interact with the simulated sea area. During the interaction, calculate the actions performed by the first underwater vehicle based on its kinematic and dynamic models. The state after And based on the current state of the first underwater vehicle, the execution action is calculated according to the reward function of the first underwater vehicle. Reward value of the process State action state and reward value Transmit to the experience replay pool, and transfer the state action state and reward value As a first sample of the first underwater vehicle;

[0038] Each second underwater vehicle executes the actions output by its respective second decision network. The system interacts with the simulated sea area, and during the interaction, the actions performed by each second underwater vehicle are calculated based on its kinematic and dynamic models. The state after And based on the current state of the second underwater vehicle, the actions to be performed by the second underwater vehicle are calculated according to its reward function. Reward value of the process State action state Reward Value Transferred to the experience replay pool;

[0039] action For action The product of k and action The state of the first underwater vehicle The input is fed into the first decision network to obtain the action, and then the action is... Transferred to the experience replay pool;

[0040] The state in the experience replay pool action state Reward Value action and actions This serves as a second sample of the second underwater vehicle.

[0041] The method for obtaining the first and second samples from the experience replay pool in the above technical solution includes the following specific steps:

[0042] Step 1: Based on ocean current data of the simulated sea area, set the mission start point, mission end point, and multiple waypoints on the navigation route from the mission start point to the mission end point. The initial position of the first underwater vehicle is used as the mission start point, and the initial position of each second underwater vehicle is set according to the mission start point.

[0043] Step 2: Take the next waypoint that the first underwater vehicle is about to reach in the navigation route as the current target waypoint of the first underwater vehicle, so that the first underwater vehicle sails toward the current target waypoint, and make N second underwater vehicles sail together with the first underwater vehicle with the same offset.

[0044] Step 3: The first underwater vehicle travels along the same route H times, and N second underwater vehicles travel together with the first underwater vehicle while maintaining the same offset. Multiple first samples from the first underwater vehicle and multiple second samples from each second underwater vehicle are obtained. The number of first samples and second samples obtained in each voyage is multiple.

[0045] In the above technical solution, after obtaining "one first sample and one second sample corresponding to each second underwater vehicle", the state in the first sample is... As the first underwater vehicle in its current state, the following judgments are made:

[0046] Determine whether the Euclidean distance l2 between the current location of the first underwater vehicle and the current target waypoint is less than the threshold m1:

[0047] If the Euclidean distance l2 is not less than the threshold m1, then determine whether either condition (1) or condition (2) is satisfied: if neither condition (1) nor condition (2) is satisfied, then proceed to obtain "one first sample and one second sample corresponding to each second underwater vehicle", that is, the first underwater vehicle and each second underwater vehicle continue to sail towards the current target waypoint; if either condition (1) or condition (2) is satisfied, then the current voyage ends.

[0048] If the Euclidean distance l2 is less than the threshold m1, then determine whether either condition (2) or condition (3) is satisfied: if either condition (2) or condition (3) is not satisfied, select the path point pointed to by the current target waypoint in the navigation route as the next current target waypoint, and proceed to obtain "one first sample and one second sample corresponding to each second underwater vehicle" for the next time; if either condition (2) or condition (3) is satisfied, then the current navigation ends; where:

[0049] Condition (1): The coordinates of the location of the first underwater vehicle in its current state are outside the simulated sea area;

[0050] Condition (2): l1 + l2 - l g >m2, where l1 is the Euclidean distance between the coordinates of the first underwater vehicle's current position and the previous target waypoint, and l2 is the Euclidean distance between the coordinates of the first underwater vehicle's current position and the current target waypoint. g This is the Euclidean distance between the previous waypoint and the current waypoint.

[0051] Condition (3): The current target waypoint is the mission endpoint, that is, the first underwater vehicle has reached the mission endpoint.

[0052] The present invention has the following advantages due to the adoption of the above technical solutions:

[0053] 1. The underwater vehicle deep reinforcement learning formation control method of the present invention utilizes communication to provide additional state awareness to the second underwater vehicle, that is, the actions performed by the first underwater vehicle. Through richer state information input, the second underwater vehicle can more accurately predict the navigation intention of the first underwater vehicle, thereby maintaining a better formation.

[0054] 2. When communication is good, the second underwater vehicle obtains the true action intention of the first underwater vehicle through communication. If there is an unexpected situation such as untimely communication, the second underwater vehicle uses the imitation learning compensator to use its output as the estimated action intention of the first underwater vehicle to assist its own action decision-making and avoid the situation of undesirable formation or being broken up. Attached Figure Description

[0055] Figure 1 This is the reinforcement learning model for underwater vehicles of the present invention;

[0056] Figure 2 A schematic diagram showing the included angle between the first and second underwater vehicles;

[0057] Figure 3 This is a schematic diagram of the formation effect in Example 9;

[0058] Figure 4 This is a schematic diagram illustrating the formation effect of the underwater vehicle deep reinforcement learning formation control method of the present invention;

[0059] Figure 5 This is a comparison diagram of the training convergence of the underwater vehicle deep reinforcement learning formation control method of the present invention and Example 9. Detailed Implementation

[0060] The following detailed description of a deep reinforcement learning formation control method for underwater vehicles according to the present invention, with reference to the accompanying drawings and embodiments, is provided in detail.

[0061] During the navigation of the first underwater vehicle, both the first and second underwater vehicles make a decision every Δt time interval. The time at which the decision is made is called the decision time, where Δt is called the decision duration, and the value of Δt ranges from

[10] . -3 10 3 In Example 2 below, Δt = 0.1s.

[0062] Example 1

[0063] like Figure 1As shown, an underwater vehicle reinforcement learning model includes: a first decision network and N second decision networks, where N≥1, and in this embodiment, N=2; each first decision network corresponds to a first underwater vehicle, and each second decision network corresponds to a second underwater vehicle. The first underwater vehicle is used to navigate according to a route, and the second underwater vehicle is used to follow the first underwater vehicle while maintaining a fixed distance and angle from it.

[0064] The input to the first decision network is the state of the first underwater vehicle at the t-th decision. The output of the first decision network is the action of the first underwater vehicle. (The first underwater vehicle needs to perform an action to reach the state of the (t+1)th decision.)

[0065] k is a constraint coefficient. The first decision network is constructed using a neural network. The first decision network includes an input layer, a first hidden layer, a second hidden layer, and an output layer. The number of nodes in the input layer is 8, the number of nodes in the first hidden layer is 400, the number of nodes in the second hidden layer is 300, and the number of nodes in the output layer is 4. The activation function of the first hidden layer and the second hidden layer is the ReLU function, and the activation function of the output layer is the Tanh function.

[0066] The state of the first underwater vehicle at decision t (x g y g ,z g (x) represents the coordinates of the current waypoint of the first underwater vehicle in the world coordinate system. l ,y l ,z l Let θ be the coordinates of the t-th decision point of the first underwater vehicle in the world coordinate system. g Let θ be the expected yaw angle when the first underwater vehicle reaches the current target waypoint. l u is the actual yaw angle when the first underwater vehicle reaches the current target waypoint. l v l w l The first underwater vehicle is located at coordinate point (x) l y l , z l At that time, in its body coordinate system, along the x-axis sequentially... l y l z l velocity in the direction, ω l The first underwater vehicle is located at coordinate point (x l y l ,z l angular velocity at time x l yl z l The directions are, in order, the x-axis, y-axis, and z-axis directions in the body coordinate system of the first underwater vehicle.

[0067] action In the body coordinate system of the first underwater vehicle, the first underwater vehicle moves along the x-axis. l y l z l Acceleration in the direction of The angular acceleration of the first underwater vehicle;

[0068] The input to the second decision network is the state of the corresponding second underwater vehicle at the t-th decision. and actions k is a constraint coefficient, and in this embodiment, k = 0.9. The output of the second decision network is the action that the second underwater vehicle needs to perform in order to follow the first underwater vehicle while maintaining a fixed distance and angle. The second decision network is constructed using a neural network, comprising an input layer, a third hidden layer, a fourth hidden layer, and an output layer. The input layer has 12 nodes, the third hidden layer has 400 nodes, the fourth hidden layer has 300 nodes, and the output layer has 4 nodes. The activation functions for the third and fourth hidden layers are both ReLU, and the activation function for the output layer is the Tanh function.

[0069] Among them, the state of the second underwater vehicle at the t-th decision (x f ,y f ,z f Let θ be the coordinates of the t-th decision point of the second underwater vehicle in the world coordinate system, and let a, b, and c be the offsets of the initial position of the second underwater vehicle relative to the initial position of the first underwater vehicle in the x, y, and z directions, respectively. f Let u be the actual yaw angle of the second underwater vehicle's t-th decision. l v f w f The following are the coordinates of the second underwater vehicle at its t-th decision along x in the body coordinate system of the second underwater vehicle. f′ y f′ z f′ velocity in the direction, ω f Let x be the angular velocity of the second underwater vehicle. f′ y f′ z f′ The directions are, in order, the x-axis, y-axis, and z-axis directions in the body coordinate system of the second underwater vehicle;

[0070] The second underwater vehicle's actions The second underwater vehicle is positioned in the body coordinate system of the second underwater vehicle at x f′ y f′ z f′ Acceleration in the direction of This represents the angular acceleration of the second underwater vehicle.

[0071] Example 2

[0072] Prepare an experience replay pool containing multiple first samples from the first underwater vehicle and multiple second samples from each second underwater vehicle.

[0073] Each first sample is: state action state and reward value state Perform actions for the first underwater vehicle The reward value of the state after the (t+1)th decision Let be the reward value for the first underwater vehicle's decision t, calculated using a reward function:

[0074]

[0075] In the formula, θ′ l The state corresponding to the first underwater vehicle at decision t+1. θ l ,like Figure 2 As shown, e α For the first underwater vehicle in state The angle between the vector pointing from the current location's coordinates to the current target waypoint and its own velocity vector. Figure 2 The "leader" in the text refers to the first underwater vehicle, k. 11 and k 12 All are coefficients, k 11 =0.5, k 12 =1.0.

[0076] Each second sample is: state action state Reward Value action and actions state Performing actions for the second underwater vehicle The state and action after the (t+1)th decision To input state into the first decision network The action obtained The product of k and the reward value Let be the reward value for the second underwater vehicle's decision t, calculated using a reward function:

[0077]

[0078] In the formula, k 21 and k 22 All are coefficients, i∈[1,N], d i For the i-th second underwater vehicle in state The distance between the current location's coordinates and its desired location's coordinates; such as Figure 2 As shown, e βi For the i-th second underwater vehicle in state The angle between the vector pointing from the current coordinate point to the desired position (the desired position is the position of the second underwater vehicle in the world coordinate system while keeping the offset between it and the first underwater vehicle constant) and its own velocity vector. Figure 2 The "follower" in this example refers to the second underwater vehicle. 21 =0.1, k 22 =1.0.

[0079] The method for obtaining the first and second samples from the experience replay pool includes the following specific steps:

[0080] S1, based on ocean current data from the simulated sea area, sets the mission start point, mission end point, and multiple waypoints A along the navigation route from the mission start point to the mission end point. i, The initial position of the first underwater vehicle is taken as the mission starting point, and the initial position of each second underwater vehicle is set according to the mission starting point.

[0081] The navigation route is: A1→A2→…→A i′ →…→A K i ′ ∈[1,K], where K is the number of all waypoints on the navigation route, A1 is the starting point of the mission, A K As the endpoint of the mission, the ocean current data in the simulated sea area are either real ocean current data or ocean current data simulated by functions.

[0082] In this embodiment, K = 10, and the coordinates of the navigation route are as follows: A1(0, 0, 0) → A2(2, 2, -1) → A3(4, 4, -2) → A4(6, 6, -3) → A5(8, 8, -3) → A6(10, 10, -3) → A7(12, 12, -3) → A8(14, 14, -2) → A9(16, 16, -1) → A 10(18, 18, 0); The initial position of the first underwater vehicle is A1(0, 0, 0), the initial position of the first underwater vehicle is (-2, 0, 0), and the initial position of the second underwater vehicle is (0, -2, 0). That is, the offsets of the first underwater vehicle are a = 2, b = 0, and c = 0, and the offsets of the second underwater vehicle are a = 0, b = 2, and c = 0.

[0083] S2, take the next waypoint that the first underwater vehicle is about to reach in the navigation route as the current target waypoint of the first underwater vehicle, make the first underwater vehicle sail towards the current target waypoint, and make N second underwater vehicles sail together with the first underwater vehicle with the offset (formation structure), and set the speed and angular velocity of each second underwater vehicle and the first underwater vehicle at their respective initial positions to 0;

[0084] S3, make the first underwater vehicle travel along the same route H times (N second underwater vehicles travel together with the offset between them and the first underwater vehicle), to obtain multiple first samples of the first underwater vehicle and multiple second samples of each second underwater vehicle, wherein the number of first samples and second samples obtained in each voyage is multiple.

[0085] The method for the first underwater vehicle to navigate according to the navigation route each time includes: sequentially obtaining multiple sets of "one first sample and one second sample corresponding to each second underwater vehicle", and after obtaining "one first sample and one second sample corresponding to each second underwater vehicle" each time, the state in the first sample is... As the first underwater vehicle in its current state, the following judgments are made:

[0086] like Figure 3 As shown, it is determined whether the Euclidean distance l2 between the coordinates of the current location of the first underwater vehicle and the current target waypoint is less than the threshold m1 (in this embodiment, m1 = 0.1):

[0087] If the Euclidean distance l2 is not less than the threshold m1, then determine whether either condition (1) or condition (2) is satisfied: if neither condition (1) nor condition (2) is satisfied, then proceed to obtain "one first sample and one second sample corresponding to each second underwater vehicle", that is, the first underwater vehicle and each second underwater vehicle continue to sail towards the current target waypoint; if either condition (1) or condition (2) is satisfied, then the current voyage ends.

[0088] If the Euclidean distance l2 is less than the threshold m1, then determine whether either condition (2) or condition (3) is satisfied: if either condition (2) or condition (3) is not satisfied, select the path point pointed to by the current target waypoint in the navigation route as the next current target waypoint, and proceed to obtain "one first sample and one second sample corresponding to each second underwater vehicle" for the next time; if either condition (2) or condition (3) is satisfied, then the current navigation ends; where:

[0089] Condition (1): The coordinates of the location of the first underwater vehicle in its current state are outside the simulated sea area;

[0090] Condition (2): l1 + l2 - l g >m2, where l1 is the Euclidean distance between the coordinates of the first underwater vehicle's current position and the previous target waypoint, and l2 is the Euclidean distance between the coordinates of the first underwater vehicle's current position and the current target waypoint. g The Euclidean distance between the previous and current target waypoints is m2 = 0.2 in this embodiment. Condition (2) is used to assess whether the first underwater vehicle deviates significantly from the preset trajectory;

[0091] Condition (3): The current target waypoint is the mission endpoint, that is, the first underwater vehicle has reached the mission endpoint.

[0092] The method for obtaining "one first sample and one second sample corresponding to each second underwater vehicle" each time:

[0093] ①The status of the first underwater vehicle The input is fed into the first decision network to obtain the action. Actions will be communicated Transmit the status of each second underwater vehicle to each second underwater vehicle. and actions The inputs are respectively fed into the second decision network corresponding to the second underwater vehicle to obtain the actions of the second underwater vehicle.

[0094] ② Enable the first underwater vehicle to interact with the simulated sea area. During the interaction, calculate the actions performed by the first underwater vehicle based on its kinematic and dynamic models. The state after And based on the current state of the first underwater vehicle, the execution action is calculated according to the reward function of the first underwater vehicle. Reward value of the process State action state and reward value Transmit to the experience replay pool, and transfer the state action state and reward value As a first sample of the first underwater vehicle.

[0095] Each second underwater vehicle executes the actions output by its respective second decision network. The system interacts with the simulated sea area, and during the interaction, the actions performed by each second underwater vehicle are calculated based on its kinematic and dynamic models. The state after And based on the current state of the second underwater vehicle, the actions to be performed by the second underwater vehicle are calculated according to its reward function. Reward value of the process State action state Reward Value Transferred to the experience replay pool;

[0096] action For action The product of k and action The state of the first underwater vehicle The input is fed into the first decision network to obtain the action, and then the action is... Transferred to the experience replay pool;

[0097] The state in the experience replay pool action state Reward Value action and actions This serves as a second sample of the second underwater vehicle.

[0098] Example 3

[0099] In Example 2, after each voyage, the first underwater vehicle executes the "training method of underwater vehicle reinforcement learning model". The training method of underwater vehicle reinforcement learning model includes steps S1 and S2. Every 2T times the training method of underwater vehicle reinforcement learning model is executed, the updated first decision network and the updated second decision network are used to perform a voyage according to the voyage route in Example 2. If the first underwater vehicle and N second underwater vehicles can maintain formation and reach the mission endpoint in 5 consecutive voyages, the training is stopped.

[0100] Training methods for reinforcement learning models of underwater vehicles include:

[0101] S1. Randomly select M first samples from one voyage of the first underwater vehicle. Set up evaluation networks Q1, Q2, Q3, Q4, and decision network P1. Update the first decision network, evaluation network Q1, and evaluation network Q2 sequentially with the M first samples. The method for updating the first decision network, evaluation network Q1, and evaluation network Q2 sequentially for each first sample is as follows:

[0102] The state in the first sample The input is fed into the first decision network to obtain the actions of the first underwater vehicle. As an action The state in the first sample and actions The value is obtained by inputting the data into the evaluation network Q1. The state in the first sample and actions The value is obtained by inputting the data into the evaluation network Q2. The state in the first sample The input is fed into decision network P1 to obtain the actions of the first underwater vehicle. As an action And add normally distributed noise.

[0103] State and actions The value is estimated by inputting it into the evaluation network Q3. State and actions The value is estimated by inputting it into the evaluation network Q4. Selection of value estimation and valuation The smaller value in the middle is used as the value estimate Based on value estimation and the reward value in the sample The value estimate was calculated. (Fujimoto S, Van Hoof H, Meger D. Addressing Function Approximation Error in Actor-Critic Methods[Z].arXiv, 2018(2018).); Based on value estimation Value estimation Value estimation state and actions Calculate the loss function and update the first decision network, evaluation network Q1, and evaluation network Q2 of the first underwater vehicle.

[0104] The first underwater vehicle corresponds to four identical evaluation networks: Evaluation Network Q1, Evaluation Network Q2, Evaluation Network Q3, and Evaluation Network Q4. The input to each evaluation network is the state and actions of the first underwater vehicle; the output of each evaluation network is a value estimate, which is used to evaluate the actions input to the evaluation network to measure whether the actions output by the first decision network can enable the first underwater vehicle to navigate along a predetermined trajectory. Each evaluation network is constructed using a neural network and includes an input layer, a fourth hidden layer, a fifth hidden layer, and an output layer. The input layer has 12 nodes, the fourth hidden layer has 400 nodes, the fifth hidden layer has 300 nodes, and the output layer has 1 node. The activation functions of the fourth and fifth hidden layers are ReLU functions.

[0105] The decision network P1 has the same structure as the first decision network.

[0106] S2, randomly select M second samples from each second underwater vehicle during a single voyage. Each second underwater vehicle is configured with an evaluation network Q5, an evaluation network Q6, an evaluation network Q7, an evaluation network Q8, and a decision network P2. The M second samples are used to update the "N second decision networks and their corresponding evaluation networks Q5 and Q6" multiple times. The update method for each second decision network and its corresponding evaluation networks Q5 and Q6 is as follows:

[0107] The state of the second underwater vehicle in a single voyage during the second decision network's journey in a second sample. and actions The input is fed into the second decision network to obtain the action. As an action The states from one second sample of each of the N second underwater vehicles and actions The value estimate is obtained by inputting the data into the evaluation network Q5 corresponding to the second decision network. (The state of each of the second underwater vehicles in one second sample) and actions (Input together into the evaluation network Q5 corresponding to the same second underwater vehicle); the states from one second sample of each of the N second underwater vehicles and actions The value estimate is obtained by inputting the data into the evaluation network Q6 corresponding to the second decision network. (The state of each of the second underwater vehicles in one second sample) and actions (Input together into the same evaluation network Q6 corresponding to the second underwater vehicle); the state in the second sample corresponding to the second decision network. and actions The input is fed into decision network P2, corresponding to the second decision network, to obtain the actions of the second underwater vehicle. As an action And add normally distributed noise;

[0108] The states from one second sample of each of the N second underwater vehicles The individual actions of N second underwater vehicles The inputs are combined with the evaluation network Q7 corresponding to the second decision network to obtain the output value estimate. (The state of each of the second underwater vehicles in the second sample) and the actions of all second underwater vehicles The data is input together into the evaluation network Q7 corresponding to one second underwater vehicle; the states from one second sample of each of the N second underwater vehicles are then processed. and the actions of all second underwater vehicles The values ​​are then input together into the evaluation network Q8 corresponding to the second decision network to obtain the value estimate. (The states in the N second samples corresponding to all the second underwater vehicles) And the actions of N second underwater vehicles The data is input together into the evaluation network Q8 corresponding to a second underwater vehicle; value estimation is then selected. and valuation The smaller value in the middle is used as the value estimate

[0109] Based on value estimation and the reward value in the sample Calculate the target value estimate (Fujimoto S, Van Hoof H, Meger D. Addressing Function Approximation Error in Actor-Critic Methods[Z].arXiv, 2018(2018).); Based on value estimation Value estimation Target value estimation The state of the second sample of N second underwater vehicles And the actions of N second underwater vehicles Calculate the loss function and update the second decision network, evaluation network Q5, and evaluation network Q6 for each second underwater vehicle;

[0110] The four evaluation networks (evaluation network Q5, evaluation network Q6, evaluation network Q7, and evaluation network Q8) corresponding to one of the second underwater vehicles have the same structure. Each evaluation network of the second underwater vehicle is constructed using a neural network, including: an input layer, a sixth hidden layer, a seventh hidden layer, and an output layer. The number of nodes in the input layer is 12×N, the number of nodes in the sixth hidden layer is 400, the number of nodes in the seventh hidden layer is 300, and the number of nodes in the output layer is 1. The activation function of the sixth and seventh hidden layers is the ReLU function.

[0111] The decision network P2 has the same structure as the second decision network (each second underwater vehicle corresponds to one decision network P2).

[0112] The update method for the first decision network, evaluation network Q1, evaluation network Q2, second decision network, evaluation network Q5, and evaluation network Q6 is based on the MATD3 method: the first decision network is updated once for every n updates to evaluation network Q1 and evaluation network Q2; the second decision network is updated once for every n updates to evaluation network Q5 and evaluation network Q6.

[0113] The specific calculation formula of the loss function used in this embodiment can be found in the paper: Fujimoto S, VanHoof H, Meger D. Addressing Function Approximation Error in Actor-Critic Methods[Z].arXiv,2018(2018).

[0114] In this embodiment, T = 100, M = 100, and n = 2.

[0115] Example 4

[0116] Based on Example 3, an underwater vehicle reinforcement learning model further includes: an imitation learning compensator corresponding to each second underwater vehicle; when the second underwater vehicle cannot obtain the actions of the first underwater vehicle via communication, the imitation learning compensator is used to replace the first decision network; the input of the imitation learning compensator is the state. Imitate the output action of the learning compensator As an action The imitation learning compensator is constructed using a neural network and includes an input layer, an eighth hidden layer, a ninth hidden layer, a tenth hidden layer, an eleventh hidden layer, and an output layer. The input layer has 8 nodes, the eighth hidden layer has 1400 nodes, the ninth hidden layer has 1000 nodes, the tenth hidden layer has 600 nodes, the eleventh hidden layer has 300 nodes, and the output layer has 4 nodes. The activation function of the eighth, ninth, tenth, and eleventh hidden layers is the ReLU function, and the activation function of the output layer is the Tanh function.

[0117] Example 5

[0118] Based on Example 4, a training method for an underwater vehicle reinforcement learning model includes: during the repeated execution of S1 and S2 in Example 3, every T executions of S2 in Example 3, each imitation learning compensator is updated 1000 times; each time the imitation learning compensator is updated, 400 first samples are randomly selected from the experience replay pool, and the states in the first samples are... Input is fed into the imitation learning compensator, and the output is the action. Based on the actions in the first sample and the action output of the imitation learning compensator Calculate the loss function and update the imitation learning compensator; where the loss function of the imitation learning compensator is the action in the first sample. and the action output of the imitation learning compensator The mean square error between them.

[0119] Example 6

[0120] Based on Example 3, a deep reinforcement learning formation control method for underwater vehicles includes: deploying a trained underwater vehicle reinforcement learning model to the mission starting point in the marine environment; maintaining communication between the second underwater vehicle (two second underwater vehicles) and the first underwater vehicle (i.e., without imitation learning compensators); the first underwater vehicle continuously outputting the current decision action using its first decision network, calculating the state of the next decision based on kinematic and dynamic models, and navigating according to the navigation route based on the action; the second underwater vehicle acquiring the current decision action of the first underwater vehicle through communication, continuously outputting the next decision action using its second decision network, and calculating the state of the next decision based on kinematic and dynamic models, thereby achieving formation navigation with the first underwater vehicle.

[0121] Example 7

[0122] A deep reinforcement learning formation control method for underwater vehicles is basically the same as that in Example 6, except that the communication between the second underwater vehicle and the first underwater vehicle is limited. The method for obtaining the action of the first underwater vehicle is as follows: the second underwater vehicle obtains the state of the first underwater vehicle through communication, and inputs the state of the first underwater vehicle into the imitation learning compensator of each second underwater vehicle in Example 5 to obtain the action of the first underwater vehicle.

[0123] Example 8

[0124] A deep reinforcement learning formation control method for underwater vehicles is basically the same as that in Example 7, except that the communication between the second underwater vehicle and the first underwater vehicle is severely restricted. The method for obtaining the action of the first underwater vehicle is as follows: the state of the first underwater vehicle before executing the action of the current decision is calculated based on the action of the previous decision, the kinematic model and the dynamic model of the first underwater vehicle. The state is then input into the imitation learning compensator of each second underwater vehicle in Example 5 to obtain the action as the action of the current decision of the first underwater vehicle.

[0125] Example 9 (for comparison)

[0126] A deep reinforcement learning-based formation control method for underwater vehicles includes: using the deep reinforcement learning method MATD3 (Addressing Function Approximation Error in Actor-Critic Methods) to achieve formation control of two second underwater vehicles and a first underwater vehicle.

[0127] In Example 6, the second underwater vehicle acquires the actions of the first underwater vehicle. After increasing the input dimension of the decision network of the second underwater vehicle, its training complexity is basically unaffected, and it can still form a formation quickly and reliably.

[0128] The formation effects of Examples 9 and 7 are as follows: Figure 3 and Figure 4 As shown.

[0129] Figure 5 This is a comparison chart showing the convergence of the training method for the underwater vehicle reinforcement learning model of the present invention in Example 3 (PC-MATD3) and Example 9 (MATD3) during training. The chart shows that... Figure 5The middle curve represents the mean of the round-average reward of the second underwater vehicle (filled with 95% confidence intervals). In the early stage of training, the navigation paths of the second underwater vehicle and the first underwater vehicle in the deep reinforcement learning formation control method of the present invention are relatively short, and the round-average reward of the second underwater vehicle is higher. As the decision network of the first underwater vehicle is updated, its navigation path gradually becomes longer, the tracking error of the second underwater vehicle gradually increases, and its round-average reward decreases. As the decision network of the second underwater vehicle is updated, the tracking error of the second underwater vehicle gradually decreases, and its round-average reward recovers and converges. As can be seen from the figure, the round-average reward of the deep reinforcement learning formation control method of the present invention is smoother than that of the comparative method, and the variance fluctuation range of the shaded area is smaller, which is more superior.

[0130] The underwater vehicle deep reinforcement learning formation control method of the present invention (which still uses an imitation learning compensator to acquire the actions of the first underwater vehicle when communication is unrestricted) and Example 9, with mean distance error, standard deviation of distance error, mean angle error, and standard deviation of angle error in formation across multiple navigation routes, are shown in Table 1. As can be seen from Table 1, after the second underwater vehicle uses the imitation learning compensator to acquire the actions of the first underwater vehicle, the mean distance error and mean angle error of the formation are reduced by approximately 10%, and the standard deviation of distance error and standard deviation of angle error are reduced by more than 30%, demonstrating superior performance and stability.

[0131] Table 1

[0132]

[0133] The communication loss probabilities of Examples 8 and 9 were set to 0.2, 0.5, and 0.8, respectively. In Example 8, the second underwater vehicle used only the actions output by the imitation learning compensator as the actions of the first underwater vehicle for formation navigation. The mean distance error and mean angle error in a navigation route are shown in Table 2.

[0134] Among them, the communication loss probability is used to describe the proportion of the process of "calculating the position and state of the first underwater vehicle by using the action output of the first underwater vehicle in Example 8" in the overall formation navigation process.

[0135] Table 2

[0136]

[0137]

[0138] As shown in Table 2, as the probability of communication loss increases, the formation error of Example 9 gradually increases, and the anti-interference ability is poor. However, in Example 8, the imitation learning compensator is used to predict the intention of the first underwater vehicle and the position of the first underwater vehicle is estimated when communication is lost. The formation is almost unaffected by the communication loss. The underwater vehicle reinforcement learning model of the present invention continues to perform stably.

[0139] The present invention has been described above by way of example. It should be noted that any simple modifications, alterations or other equivalent substitutions that can be made by those skilled in the art without creative effort without departing from the core of the present invention fall within the protection scope of the present invention.

Claims

1. An underwater vehicle reinforcement learning model, characterized in that, Comprise: A first decision network and N second decision networks, wherein N≥1, each first decision network corresponds to a first underwater vehicle, each second decision network corresponds to a second underwater vehicle, the first underwater vehicle is used to navigate according to the route, and the second underwater vehicle is used to follow the first underwater vehicle under the premise of maintaining a fixed distance and angle with the first underwater vehicle; The first underwater vehicle and the second underwater vehicle make a decision every △t time interval respectively; An input to the first decision network is a state of the first underwater vehicle at the tth decision An output of the first decision network is an action of the first underwater vehicle The input of the second decision network is the state of the corresponding second underwater vehicle at the tth decision and action k is a constraint coefficient, and the output of the second decision network is the action of the second underwater vehicle The first underwater vehicle at the tth decision-making state (x g , y g , z g ) is the coordinate point of the current target waypoint of the first underwater vehicle in the world coordinate system, (x l , y l , z l ) is the coordinate point of the first underwater vehicle at the tth decision-making position in the world coordinate system, θ g is the expected yaw angle of the first underwater vehicle when reaching the current target waypoint, θ l is the actual yaw angle of the first underwater vehicle when reaching the current target waypoint, u l , v l , w l are respectively the velocities of the first underwater vehicle in the x l , y l , z l directions in its body coordinate system when located at the coordinate point (x l , y l , z l ), ω l is the angular velocity of the first underwater vehicle when located at the coordinate point (x l , y l , z l ), and x l , y l , z l are respectively the x-axis, y-axis and z-axis directions of the body coordinate system of the first underwater vehicle; Actions are the accelerations of the first underwater vehicle in the x l , y l , z l directions in the body coordinate system of the first underwater vehicle, respectively, is the angular acceleration of the first underwater vehicle; The second underwater vehicle at the tth decision-making state (x f , y f , z f ) is a coordinate point of the position of the tth decision-making of the second underwater vehicle under the world coordinate system, a, b, c are offset amounts of the coordinate point of the initial position of the second underwater vehicle under the world coordinate system relative to the coordinate point of the initial position of the first underwater vehicle in x, y, z directions respectively, θ f is the actual yaw angle of the tth decision-making of the second underwater vehicle, u f , v f , w f are respectively the velocities of the tth decision-making of the second underwater vehicle along x f ', y f ', z f ' directions under the body coordinate system of the second underwater vehicle, ω f is the angular velocity of the second underwater vehicle, x f ', y f ', z f ' directions are respectively the x axis, y axis and z axis directions under the body coordinate system of the second underwater vehicle; Actions of the second underwater vehicle are the accelerations of the second underwater vehicle in the x f ′, y f ′, z f ′ directions, respectively, in the body coordinate system of the second underwater vehicle, is the angular acceleration of the second underwater vehicle.

2. The underwater vehicle reinforcement learning model of claim 1, wherein, The first decision network is constructed by using a neural network, and the second decision network is constructed by using a neural network.

3. The underwater vehicle reinforcement learning model according to claim 1 or 2, wherein, Also include: one imitation learning compensator per second underwater vehicle, the input of the imitation learning compensator being a state the output of the imitation learning compensator being an action as an action The imitation learning compensator is constructed using a neural network.

4. The method of claim 1-3, wherein, Every 2T times of underwater vehicle reinforcement learning model training method, then use the updated first decision network and the updated second decision network to navigate once; If the first underwater vehicle and the N second underwater vehicles can maintain the formation to reach the task end point for at least 5 consecutive times, stop training; Each first sample is: a state an action a state and a reward value a state performing an action for the first underwater vehicle a reward value at the state of the t+1 decision a reward value for the first underwater vehicle for the t decision, the reward value being calculated by a reward function: In the formula, θ′ l The state corresponding to the first underwater vehicle at decision t+1. θ l e α For the first underwater vehicle in state The angle between the vector pointing from the current location's coordinates to the current target waypoint and its own velocity vector, k 11 and k 12 All are coefficients; Each second sample is: state action state reward value action and action state performing action for the second underwater vehicle after state at the t+1 decision, action inputting state to the first decision network resulting action and k, reward value reward value for the second underwater vehicle for the t decision, reward value is calculated by reward function: wherein k 21 and k 22 are coefficients, i∈[1, N], d i is the distance between the coordinate point of the i-th second underwater vehicle's position in state and the coordinate point of its desired position; e βi is the angle between the vector from the coordinate point of the i-th second underwater vehicle's position in state to the coordinate point of its desired position and its own velocity vector. The training method of underwater vehicle reinforcement learning model comprises S1 and S2: S1, randomly extract M first samples in a first underwater vehicle navigation, set evaluation network Q1, evaluation network Q2, evaluation network Q3, evaluation network Q4 and decision network P1, update the first decision network, evaluation network Q1 and evaluation network Q2 in turn with M first samples, and the method of updating the first decision network, evaluation network Q1 and evaluation network Q2 in turn with each first sample is as follows: the state in the first sample to the first decision network to obtain an action by the first underwater vehicle as the action the state in the first sample and the action to the evaluation network Q1 to obtain a value estimate the state in the first sample and the action to the evaluation network Q2 to obtain a value estimate the state in the first sample to the first decision network to obtain an action by the first underwater vehicle as the action input to the evaluation network Q3 to obtain a value estimate and action input to the evaluation network Q4 to obtain a value estimate input to the evaluation network Q3 to obtain a value estimate and action input to the evaluation network Q4 to obtain a value estimate selecting the smaller value estimate from the value estimate and the value estimate as the value estimate computing the value estimate from the value estimate and the reward value in the sample computing the value estimate the value estimate the value estimate state and action computing a loss function and updating the first decision network, the evaluation network Q1, and the evaluation network Q2 of the first underwater vehicle S2, randomly extract M second samples in each second underwater vehicle navigation, set an evaluation network Q5, an evaluation network Q6, an evaluation network Q7, an evaluation network Q8 and a decision network P2 for each second underwater vehicle, update "N second decision networks and their respective evaluation networks Q5 and evaluation networks Q6" multiple times, and the method of updating each second decision network and its corresponding evaluation network Q5 and evaluation network Q6 is as follows: the state in one second sample of the N second underwater vehicles and action input into the second decision network, obtaining action as action the state in one second sample of the N second underwater vehicles and action input into the evaluation network Q5 corresponding to the second decision network, obtaining value estimate the state in one second sample of the N second underwater vehicles and action input into the evaluation network Q6 corresponding to the second decision network, obtaining value estimate the state in one second sample of the second decision network corresponding to the second underwater vehicle and action input into the decision network P2 corresponding to the second decision network, obtaining action of the second underwater vehicle as action the state of each of the N second underwater vehicles in one second sample and the action of each of the N second underwater vehicles together into the evaluation network Q7 corresponding to the second decision network to obtain an output value estimate the state of each of the N second underwater vehicles in one second sample and the action of all of the second underwater vehicles together into the evaluation network Q8 corresponding to the second decision network to obtain a value estimate selecting the smaller of the value estimate and the value estimate as the value estimate According to the value estimate And the reward value in the sample The target value estimate is calculated According to the value estimate The value estimate The target value estimate The state in the second sample of the N second underwater vehicles And the action of the N second underwater vehicles The loss function is calculated, and the second decision network, the evaluation network Q5, and the evaluation network Q6 of each second underwater vehicle are updated.

5. The training method of claim 4, wherein, The updating method of the first decision network, the evaluation network Q1, the evaluation network Q2, the second decision network, the evaluation network Q5 and the evaluation network Q6 is based on the MATD3 method: update the first decision network once every n times of updating the evaluation network Q1 and the evaluation network Q2; Update the second decision network once every n times of updating the evaluation network Q5 and the evaluation network Q6.

6. The training method of claim 5, wherein, The method of obtaining "a first sample and a second sample corresponding to each second underwater vehicle" each time: the state of the first underwater vehicle is input into a first decision network to obtain an action the action is transmitted to each second underwater vehicle via communication, the state of each second underwater vehicle and the action are input into a second decision network corresponding to the second underwater vehicle respectively to obtain an action of the second underwater vehicle ​​​​ ② make the first underwater vehicle interact with the simulated sea area, and in the interaction process, the state of the first underwater vehicle after performing the action is calculated according to the kinematic model and the dynamic model of the first underwater vehicle and as the current state of the first underwater vehicle, the reward value of performing the action is calculated according to the reward function of the first underwater vehicle state action state and reward value as a first sample of the first underwater vehicle;​​ causing each second underwater vehicle to perform an action output by the respective second decision network interacting with the simulated sea area, and during the interaction, calculating, according to a kinematic model and a dynamic model of each second underwater vehicle, a state of the second underwater vehicle after performing the action and taking the state as a current state of the second underwater vehicle, calculating, according to a reward function of the second underwater vehicle, a reward value of the second underwater vehicle performing the action process ​ action for the action and the product of k, action state of the first underwater vehicle input into the first decision network to obtain the action; state action state reward value action and action as a second sample of the second underwater vehicle of the station.

7. The training method of claim 6, wherein, Make the first underwater vehicle navigate according to the same navigation route H times, and the N second underwater vehicles navigate together with the offset between the first underwater vehicle, obtain multiple first samples of the first underwater vehicle and multiple second samples of each second underwater vehicle, wherein the number of first samples and second samples obtained each time is multiple.

8. The training method of claim 7, wherein, After obtaining "one first sample and one second sample corresponding to each second underwater vehicle" each time, the state in the first sample is As the current state of the first underwater vehicle, the following judgment is made: Determine whether the Euclidean distance l2 between the coordinate point of the location of the first underwater vehicle in the current state and the current target navigation point is less than the threshold value m1: If the Euclidean distance l2 is not less than the threshold m1, it is judged whether any one of condition (1) and condition (2) is satisfied: if neither condition (1) nor condition (2) is satisfied, the next "one first sample and one second sample corresponding to each second underwater vehicle" is obtained; if any one of condition (1) and condition (2) is satisfied, the current voyage ends; If the Euclidean distance l2 is less than the threshold m1, it is judged whether any one of condition (2) and condition (3) is satisfied: if any one of condition (2) and condition (3) is not satisfied, a path point pointed to by the current target waypoint in the voyage route is selected as a next current target waypoint, and the next "one first sample and one second sample corresponding to each second underwater vehicle" is obtained; if any one of condition (2) and condition (3) is satisfied, the current voyage ends; wherein: Condition (1): the coordinate point of the position of the first underwater vehicle in the current state is out of the simulation sea area range; Condition (2): l1 + l2 - l g > m2, wherein, l1 is the Euclidean distance between the coordinate point of the location of the first underwater vehicle in the current state and the last target navigation point, l2 is the Euclidean distance between the coordinate point of the location of the first underwater vehicle in the current state and the current target navigation point, l g is the Euclidean distance between the last target navigation point and the current target navigation point; Condition (3): the current target waypoint is the task end point.

Images