Underwater vehicle depth reinforcement learning formation control method

Abstract

The application discloses a kind of underwater vehicle depth reinforcement learning formation control method based on imitative learning compensator, underwater vehicle reinforcement learning model includes: a first decision network and N second decision networks, the input of first decision network is the state of first underwater vehicle, and the output of first underwater vehicle is the action of first underwater vehicle, the input of second decision network is the state and action of corresponding second underwater vehicle, and the output of the second underwater vehicle is the action of the second underwater vehicle, when second underwater vehicle cannot obtain the action of first underwater vehicle, imitative learning compensator is used instead of first decision network, and the input is state, and the output is action.The underwater vehicle depth reinforcement learning formation control method of the application provides state perception for second underwater vehicle, so that second underwater vehicle accurately predicts the navigation intention of first underwater vehicle, and maintains a better formation.

Description

Technical Field

[0001] This invention belongs to the field of underwater vehicle control technology, specifically relating to a deep reinforcement learning formation control method for underwater vehicles. Background Technology

[0002] As a new type of underwater robot, underwater vehicles have received extensive research attention both domestically and internationally in recent years. Equipped with sensors, navigation systems, and power units, underwater vehicles can autonomously complete tasks based on pre-programmed instructions or real-time data. Therefore, underwater vehicles are characterized by high autonomy, high safety, high efficiency, and low cost.

[0003] In recent years, with the continuous deepening of ocean exploration, a single underwater vehicle can no longer meet the needs of exploration, and forming a fleet of multiple underwater vehicles has become an important issue. Deploying multiple underwater vehicles simultaneously can expand the coverage area in the water at the same time, facilitating three-dimensional and three-dimensional exploration.

[0004] However, the attitude of current underwater vehicles is easily affected by ocean currents and waves, and their navigation trajectories differ significantly from the preset trajectories. The method of pre-setting trajectories for each underwater vehicle to maintain formation is difficult to overcome this deviation. This can easily lead to unsatisfactory formations or even breakup of the formation due to different veer directions of the underwater vehicles.

[0005] Furthermore, when using a master-slave topology formation for multiple underwater vehicles, the state perception between the underwater vehicles is easily affected by factors such as communication delays and interruptions. This makes it difficult to ensure that the control algorithm can obtain the motion state information of the first underwater vehicle in a timely manner, which makes it difficult to maintain a stable formation.

[0006] In summary, there is an urgent need for a deep reinforcement learning-based formation control method for underwater vehicles to address the problems of unsatisfactory formations and spatiotemporal asynchrony that arise when underwater vehicles are controlled by pre-set trajectories. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a reinforcement learning model for underwater vehicles.

[0008] Another objective of this invention is to provide a training method for a reinforcement learning model of an underwater vehicle.

[0009] The present invention is achieved through the following technical solutions.

[0010] An underwater vehicle reinforcement learning model includes: a first decision network and N second decision networks, where N≥1. Each first decision network corresponds to a first underwater vehicle, and each second decision network corresponds to a second underwater vehicle. The first underwater vehicle navigates along a route, and the second underwater vehicle follows the first underwater vehicle while maintaining a fixed distance and angle. The first and second underwater vehicles each make a decision every Δt time interval.

[0011] The input to the first decision network is the state of the first underwater vehicle at the t-th decision. The output of the first decision network is the action of the first underwater vehicle.

[0012] The input to the second decision network is the state of the corresponding second underwater vehicle at the t-th decision. and actions k is a constraint coefficient, and the output of the second decision network is the action of the second underwater vehicle.

[0013] The state of the first underwater vehicle at decision t (x g ,y g ,z g (x) represents the coordinates of the current waypoint of the first underwater vehicle in the world coordinate system. l ,y l , z l Let θ be the coordinates of the t-th decision point of the first underwater vehicle in the world coordinate system. g Let θ be the expected yaw angle when the first underwater vehicle reaches the current target waypoint. l u is the actual yaw angle when the first underwater vehicle reaches the current target waypoint. l v l w l The first underwater vehicle is located at coordinate point (x) l y l , z l At that time, in its body coordinate system, along x... l y l z l velocity in the direction, ω l The first underwater vehicle is located at coordinate point (x l ,y l ,z l angular velocity at time x l y k z k The directions are, in order, the x-axis, y-axis, and z-axis directions in the body coordinate system of the first underwater vehicle.

[0014] action In the body coordinate system of the first underwater vehicle, the first underwater vehicle moves along the x-axis. k y k z k Acceleration in the direction of The angular acceleration of the first underwater vehicle;

[0015] The state of the second underwater vehicle at decision t (x f ,y f ,z f Let θ be the coordinates of the t-th decision point of the second underwater vehicle in the world coordinate system, and let a, b, and c be the offsets of the initial position of the second underwater vehicle relative to the initial position of the first underwater vehicle in the x, y, and z directions, respectively. f Let u be the actual yaw angle of the second underwater vehicle's t-th decision. f v f w f The following are the coordinates of the second underwater vehicle at its t-th decision along x in the body coordinate system of the second underwater vehicle. f′ y f′ z f′ velocity in the direction, ω f Let x be the angular velocity of the second underwater vehicle. f′ y f′ z f′ The directions are, in order, the x-axis, y-axis, and z-axis directions in the body coordinate system of the second underwater vehicle;

[0016] The second underwater vehicle's actions The second underwater vehicle is positioned in the body coordinate system of the second underwater vehicle at x f′ y f′ z f′ Acceleration in the direction of This represents the angular acceleration of the second underwater vehicle.

[0017] In the above technical solution, the first decision network is constructed using a neural network, and the second decision network is constructed using a neural network.

[0018] The above technical solution also includes: a simulation learning compensator corresponding to each second underwater vehicle, wherein the input of the simulation learning compensator is the state. Imitate the output action of the learning compensator As an action The imitation learning compensator is constructed using a neural network.

[0019] The training method for the underwater vehicle reinforcement learning model described above involves performing a voyage once every 2T training cycles using the updated first decision network and the updated second decision network. If the first underwater vehicle and N second underwater vehicles can maintain formation and reach the mission endpoint in at least 5 consecutive voyages, then the training is stopped.

[0020] Each first sample is: state action state and reward value state Perform actions for the first underwater vehicle The reward value of the state after the (t+1)th decision Let be the reward value for the first underwater vehicle's decision t, calculated using a reward function:

[0021]

[0022] In the formula, θ′ l The state corresponding to the first underwater vehicle at decision t+1. θ l e α For the first underwater vehicle in state The angle between the vector pointing from the current location's coordinates to the current target waypoint and its own velocity vector, k 11 and k 12 All are coefficients;

[0023] Each second sample is: state action state Reward Value action and actions state Performing actions for the second underwater vehicle The state and action after the (t+1)th decision To input state into the first decision network The action obtained The product of k and the reward value Let be the reward value for the second underwater vehicle's decision t, calculated using a reward function:

[0024]

[0025] In the formula, k 21 and k 22 All are coefficients, i∈[1,N], d i For the i-th second underwater vehicle in state The distance between the current location's coordinates and its desired location's coordinates; e βi For the i-th second underwater vehicle in state The angle between the vector pointing from the current coordinate point to the desired position and its own velocity vector;

[0026] The training methods for reinforcement learning models of underwater vehicles include S1 and S2:

[0027] S1. Randomly select M first samples from one voyage of the first underwater vehicle. Set up evaluation networks Q1, Q2, Q3, Q4, and decision network P1. Update the first decision network, evaluation network Q1, and evaluation network Q2 sequentially with the M first samples. The method for updating the first decision network, evaluation network Q1, and evaluation network Q2 sequentially for each first sample is as follows:

[0028] The state in the first sample The input is fed into the first decision network to obtain the actions of the first underwater vehicle. As an action The state in the first sample and actions The value is obtained by inputting the data into the evaluation network Q1. The state in the first sample and actions The value is obtained by inputting the data into the evaluation network Q2. The state in the first sample The input is fed into decision network P1 to obtain the actions of the first underwater vehicle. As an action

[0029] State and actions The value is estimated by inputting it into the evaluation network Q3. State and actions The value is estimated by inputting it into the evaluation network Q4. Selection of value estimation and valuation The smaller value in the middle is used as the value estimate Based on value estimation and the reward value in the sample The value estimate was calculated. Based on value estimation Value estimation Value estimation state and actions Calculate the loss function and update the first decision network, evaluation network Q1, and evaluation network Q2 of the first underwater vehicle;

[0030] S2, randomly select M second samples from each second underwater vehicle during a single voyage. Each second underwater vehicle is configured with an evaluation network Q5, an evaluation network Q6, an evaluation network Q7, an evaluation network Q8, and a decision network P2. The "N second decision networks and their corresponding evaluation networks Q5 and Q6" are updated multiple times. The update method for each second decision network and its corresponding evaluation networks Q5 and Q6 is as follows:

[0031] The state of the second underwater vehicle in a single voyage during the second decision network's journey in a second sample. and actions The input is fed into the second decision network to obtain the action. As an action The states from one second sample of each of the N second underwater vehicles and actions The value estimate is obtained by inputting the data into the evaluation network Q5 corresponding to the second decision network. The states from one second sample of each of the N second underwater vehicles and actions The value estimate is obtained by inputting the data into the evaluation network Q6 corresponding to the second decision network. The state in the second sample corresponding to the second decision network and actions The input is fed into decision network P2, corresponding to the second decision network, to obtain the actions of the second underwater vehicle. As an action

[0032] The states from one second sample of each of the N second underwater vehicles The individual actions of N second underwater vehicles The inputs are combined with the evaluation network Q7 corresponding to the second decision network to obtain the output value estimate. The states from one second sample of each of the N second underwater vehicles and the actions of all second underwater vehicles The values ​​are then input together into the evaluation network Q8 corresponding to the second decision network to obtain the value estimate. Selection of value estimation and valuation The smaller value in the middle is used as the value estimate

[0033] Based on value estimation and the reward value in the sample Calculate the target value estimate Based on value estimation Value estimation Target value estimation The state of the second sample of N second underwater vehicles And the actions of N second underwater vehicles Calculate the loss function and update the second decision network, evaluation network Q5, and evaluation network Q6 for each second underwater vehicle.

[0034] In the above technical solution, the first decision network, evaluation network Q1, evaluation network Q2, second decision network, evaluation network Q5, and evaluation network Q6 are updated according to the MATD3 method: the first decision network is updated once for every n updates to evaluation network Q1 and evaluation network Q2; the second decision network is updated once for every n updates to evaluation network Q5 and evaluation network Q6.

[0035] In the above technical solution, the method for obtaining "one first sample and one second sample corresponding to each second underwater vehicle" each time is as follows:

[0036] ①The status of the first underwater vehicle The input is fed into the first decision network to obtain the action. Actions will be communicated Transmit the status of each second underwater vehicle to each second underwater vehicle. and actions The inputs are respectively fed into the second decision network corresponding to the second underwater vehicle to obtain the actions of the second underwater vehicle.

[0037] ② Enable the first underwater vehicle to interact with the simulated sea area. During the interaction, calculate the actions performed by the first underwater vehicle based on its kinematic and dynamic models. The state after And based on the current state of the first underwater vehicle, the execution action is calculated according to the reward function of the first underwater vehicle. Reward value of the process State action state and reward value Transmit to the experience replay pool, and transfer the state action state and reward value As a first sample of the first underwater vehicle;

[0038] Each second underwater vehicle executes the actions output by its respective second decision network. The system interacts with the simulated sea area, and during the interaction, the actions performed by each second underwater vehicle are calculated based on its kinematic and dynamic models. The state after And based on the current state of the second underwater vehicle, the actions to be performed by the second underwater vehicle are calculated according to its reward function. Reward value of the process State action state Reward Value Transferred to the experience replay pool;

[0039] action For action The product of k and action The state of the first underwater vehicle The input is fed into the first decision network to obtain the action, and then the action is... Transferred to the experience replay pool;

[0040] The state in the experience replay pool action state Reward Value action and actions This serves as a second sample of the second underwater vehicle.

[0041] The method for obtaining the first and second samples from the experience replay pool in the above technical solution includes the following specific steps:

[0042] Step 1: Based on ocean current data of the simulated sea area, set the mission start point, mission end point, and multiple waypoints on the navigation route from the mission start point to the mission end point. The initial position of the first underwater vehicle is used as the mission start point, and the initial position of each second underwater vehicle is set according to the mission start point.

[0043] Step 2: Take the next waypoint that the first underwater vehicle is about to reach in the navigation route as the current target waypoint of the first underwater vehicle, so that the first underwater vehicle sails toward the current target waypoint, and make N second underwater vehicles sail together with the first underwater vehicle with the same offset.

[0044] Step 3: The first underwater vehicle travels along the same route H times, and N second underwater vehicles travel together with the first underwater vehicle while maintaining the same offset. Multiple first samples from the first underwater vehicle and multiple second samples from each second underwater vehicle are obtained. The number of first samples and second samples obtained in each voyage is multiple.

[0045] In the above technical solution, after obtaining "one first sample and one second sample corresponding to each second underwater vehicle", the state in the first sample is... As the first underwater vehicle in its current state, the following judgments are made:

[0046] Determine whether the Euclidean distance l2 between the current location of the first underwater vehicle and the current target waypoint is less than the threshold m1:

[0047] If the Euclidean distance l2 is not less than the threshold m1, then determine whether either condition (1) or condition (2) is satisfied: if neither condition (1) nor condition (2) is satisfied, then proceed to obtain "one first sample and one second sample corresponding to each second underwater vehicle", that is, the first underwater vehicle and each second underwater vehicle continue to sail towards the current target waypoint; if either condition (1) or condition (2) is satisfied, then the current voyage ends.

[0048] If the Euclidean distance l2 is less than the threshold m1, then determine whether either condition (2) or condition (3) is satisfied: if either condition (2) or condition (3) is not satisfied, select the path point pointed to by the current target waypoint in the navigation route as the next current target waypoint, and proceed to obtain "one first sample and one second sample corresponding to each second underwater vehicle" for the next time; if either condition (2) or condition (3) is satisfied, then the current navigation ends; where:

[0049] Condition (1): The coordinates of the location of the first underwater vehicle in its current state are outside the simulated sea area;

[0050] Condition (2): l1 + l2 - l g >m2, where l1 is the Euclidean distance between the coordinates of the first underwater vehicle's current position and the previous target waypoint, and l2 is the Euclidean distance between the coordinates of the first underwater vehicle's current position and the current target waypoint. g This is the Euclidean distance between the previous waypoint and the current waypoint.

[0051] Condition (3): The current target waypoint is the mission endpoint, that is, the first underwater vehicle has reached the mission endpoint.

[0052] The present invention has the following advantages due to the adoption of the above technical solutions:

[0053] 1. The underwater vehicle deep reinforcement learning formation control method of the present invention utilizes communication to provide additional state awareness to the second underwater vehicle, that is, the actions performed by the first underwater vehicle. Through richer state information input, the second underwater vehicle can more accurately predict the navigation intention of the first underwater vehicle, thereby maintaining a better formation.

[0054] 2. When communication is good, the second underwater vehicle obtains the true action intention of the first underwater vehicle through communication. If there is an unexpected situation such as untimely communication, the second underwater vehicle uses the imitation learning compensator to use its output as the estimated action intention of the first underwater vehicle to assist its own action decision-making and avoid the situation of undesirable formation or being broken up. Attached Figure Description

[0055] Figure 1 This is the reinforcement learning model for underwater vehicles of the present invention;

[0056] Figure 2 A schematic diagram showing the included angle between the first and second underwater vehicles;

[0057] Figure 3 This is a schematic diagram of the formation effect in Example 9;

[0058] Figure 4 This is a schematic diagram illustrating the formation effect of the underwater vehicle deep reinforcement learning formation control method of the present invention;

[0059] Figure 5 This is a comparison diagram of the training convergence of the underwater vehicle deep reinforcement learning formation control method of the present invention and Example 9. Detailed Implementation

[0060] The following detailed description of a deep reinforcement learning formation control method for underwater vehicles according to the present invention, with reference to the accompanying drawings and embodiments, is provided in detail.

[0061] During the navigation of the first underwater vehicle, both the first and second underwater vehicles make a decision every Δt time interval. The time at which the decision is made is called the decision time, where Δt is called the decision duration, and the value of Δt ranges from

[10] . -3 10 3 In Example 2 below, Δt = 0.1s.

[0062] Example 1

[0063] like Figure 1As shown, an underwater vehicle reinforcement learning model includes: a first decision network and N second decision networks, where N≥1, and in this embodiment, N=2; each first decision network corresponds to a first underwater vehicle, and each second decision network corresponds to a second underwater vehicle. The first underwater vehicle is used to navigate according to a route, and the second underwater vehicle is used to follow the first underwater vehicle while maintaining a fixed distance and angle from it.

[0064] The input to the first decision network is the state of the first underwater vehicle at the t-th decision. The output of the first decision network is the action of the first underwater vehicle. (The first underwater vehicle needs to perform an action to reach the state of the (t+1)th decision.)

[0065] k is a constraint coefficient. The first decision network is constructed using a neural network. The first decision network includes an input layer, a first hidden layer, a second hidden layer, and an output layer. The number of nodes in the input layer is 8, the number of nodes in the first hidden layer is 400, the number of nodes in the second hidden layer is 300, and the number of nodes in the output layer is 4. The activation function of the first hidden layer and the second hidden layer is the ReLU function, and the activation function of the output layer is the Tanh function.

[0066] The state of the first underwater vehicle at decision t (x g y g ,z g (x) represents the coordinates of the current waypoint of the first underwater vehicle in the world coordinate system. l ,y l ,z l Let θ be the coordinates of the t-th decision point of the first underwater vehicle in the world coordinate system. g Let θ be the expected yaw angle when the first underwater vehicle reaches the current target waypoint. l u is the actual yaw angle when the first underwater vehicle reaches the current target waypoint. l v l w l The first underwater vehicle is located at coordinate point (x) l y l , z l At that time, in its body coordinate system, along x... l y l z l velocity in the direction, ω l The first underwater vehicle is located at coordinate point (x l y l ,z l angular velocity at time x l yl z l The directions are, in order, the x-axis, y-axis, and z-axis directions in the body coordinate system of the first underwater vehicle.

[0067] action In the body coordinate system of the first underwater vehicle, the first underwater vehicle moves along the x-axis. l y l z l Acceleration in the direction of The angular acceleration of the first underwater vehicle;

[0068] The input to the second decision network is the state of the corresponding second underwater vehicle at the t-th decision. and actions k is a constraint coefficient, and in this embodiment, k = 0.9. The output of the second decision network is the action that the second underwater vehicle needs to perform in order to follow the first underwater vehicle while maintaining a fixed distance and angle. The second decision network is constructed using a neural network, comprising an input layer, a third hidden layer, a fourth hidden layer, and an output layer. The input layer has 12 nodes, the third hidden layer has 400 nodes, the fourth hidden layer has 300 nodes, and the output layer has 4 nodes. The activation functions for the third and fourth hidden layers are both ReLU, and the activation function for the output layer is the Tanh function.

[0069] Among them, the state of the second underwater vehicle at the t-th decision (x f ,y f ,z f Let θ be the coordinates of the t-th decision point of the second underwater vehicle in the world coordinate system, and let a, b, and c be the offsets of the initial position of the second underwater vehicle relative to the initial position of the first underwater vehicle in the x, y, and z directions, respectively. f Let u be the actual yaw angle of the second underwater vehicle's t-th decision. l v f w f The following are the coordinates of the second underwater vehicle at its t-th decision along x in the body coordinate system of the second underwater vehicle. f′ y f′ z f′ velocity in the direction, ω f Let x be the angular velocity of the second underwater vehicle. f′ y f′ z f′ The directions are, in order, the x-axis, y-axis, and z-axis directions in the body coordinate system of the second underwater vehicle;

[0070] The second underwater vehicle's actions The second underwater vehicle is positioned in the body coordinate system of the second underwater vehicle at x f′ y f′ z f′ Acceleration in the direction of This represents the angular acceleration of the second underwater vehicle.

[0071] Example 2

[0072] Prepare an experience replay pool containing multiple first samples from the first underwater vehicle and multiple second samples from each second underwater vehicle.

[0073] Each first sample is: state action state and reward value state Perform actions for the first underwater vehicle The reward value of the state after the (t+1)th decision Let be the reward value for the first underwater vehicle's decision t, calculated using a reward function:

[0074]

[0075] In the formula, θ′ l The state corresponding to the first underwater vehicle at decision t+1. θ l ,like Figure 2 As shown, e α For the first underwater vehicle in state The angle between the vector pointing from the current location's coordinates to the current target waypoint and its own velocity vector. Figure 2 The "leader" in the text refers to the first underwater vehicle, k. 11 and k 12 All are coefficients, k 11 =0.5, k 12 =1.0.

[0076] Each second sample is: state action state Reward Value action and actions state Performing actions for the second underwater vehicle The state and action after the (t+1)th decision To input state into the first decision network The action obtained The product of k and the reward value Let be the reward value for the second underwater vehicle's decision t, calculated using a reward function:

[0077]

[0078] In the formula, k 21 and k 22 All are coefficients, i∈[1,N], d i For the i-th second underwater vehicle in state The distance between the current location's coordinates and its desired location's coordinates; such as Figure 2 As shown, e βi For the i-th second underwater vehicle in state The angle between the vector pointing from the current coordinate point to the desired position (the desired position is the position of the second underwater vehicle in the world coordinate system while keeping the offset between it and the first underwater vehicle constant) and its own velocity vector. Figure 2 The "follower" in this example refers to the second underwater vehicle. 21 =0.1, k 22 =1.0.

[0079] The method for obtaining the first and second samples from the experience replay pool includes the following specific steps:

[0080] S1, based on ocean current data from the simulated sea area, sets the mission start point, mission end point, and multiple waypoints A along the navigation route from the mission start point to the mission end point. i, The initial position of the first underwater vehicle is taken as the mission starting point, and the initial position of each second underwater vehicle is set according to the mission starting point.

[0081] The navigation route is: A1→A2→…→A i′ →…→A K i ′ ∈[1,K], where K is the number of all waypoints on the navigation route, A1 is the starting point of the mission, A K As the endpoint of the mission, the ocean current data in the simulated sea area are either real ocean current data or ocean current data simulated by functions.

[0082] In this embodiment, K = 10, and the coordinates of the navigation route are as follows: A1(0, 0, 0) → A2(2, 2, -1) → A3(4, 4, -2) → A4(6, 6, -3) → A5(8, 8, -3) → A6(10, 10, -3) → A7(12, 12, -3) → A8(14, 14, -2) → A9(16, 16, -1) → A 10(18, 18, 0); The initial position of the first underwater vehicle is A1(0, 0, 0), the initial position of the first underwater vehicle is (-2, 0, 0), and the initial position of the second underwater vehicle is (0, -2, 0). That is, the offsets of the first underwater vehicle are a = 2, b = 0, and c = 0, and the offsets of the second underwater vehicle are a = 0, b = 2, and c = 0.

[0083] S2, take the next waypoint that the first underwater vehicle is about to reach in the navigation route as the current target waypoint of the first underwater vehicle, make the first underwater vehicle sail towards the current target waypoint, and make N second underwater vehicles sail together with the first underwater vehicle with the offset (formation structure), and set the speed and angular velocity of each second underwater vehicle and the first underwater vehicle at their respective initial positions to 0;

[0084] S3, make the first underwater vehicle travel along the same route H times (N second underwater vehicles travel together with the offset between them and the first underwater vehicle), to obtain multiple first samples of the first underwater vehicle and multiple second samples of each second underwater vehicle, wherein the number of first samples and second samples obtained in each voyage is multiple.

[0085] The method for the first underwater vehicle to navigate according to the navigation route each time includes: sequentially obtaining multiple sets of "one first sample and one second sample corresponding to each second underwater vehicle", and after obtaining "one first sample and one second sample corresponding to each second underwater vehicle" each time, the state in the first sample is... As the first underwater vehicle in its current state, the following judgments are made:

[0086] like Figure 3 As shown, it is determined whether the Euclidean distance l2 between the coordinates of the current location of the first underwater vehicle and the current target waypoint is less than the threshold m1 (in this embodiment, m1 = 0.1):

[0087] If the Euclidean distance l2 is not less than the threshold m1, then determine whether either condition (1) or condition (2) is satisfied: if neither condition (1) nor condition (2) is satisfied, then proceed to obtain "one first sample and one second sample corresponding to each second underwater vehicle", that is, the first underwater vehicle and each second underwater vehicle continue to sail towards the current target waypoint; if either condition (1) or condition (2) is satisfied, then the current voyage ends.

[0088] If the Euclidean distance l2 is less than the threshold m1, then determine whether either condition (2) or condition (3) is satisfied: if either condition (2) or condition (3) is not satisfied, select the path point pointed to by the current target waypoint in the navigation route as the next current target waypoint, and proceed to obtain "one first sample and one second sample corresponding to each second underwater vehicle" for the next time; if either condition (2) or condition (3) is satisfied, then the current navigation ends; where:

[0089] Condition (1): The coordinates of the location of the first underwater vehicle in its current state are outside the simulated sea area;

[0090] Condition (2): l1 + l2 - l g >m2, where l1 is the Euclidean distance between the coordinates of the first underwater vehicle's current position and the previous target waypoint, and l2 is the Euclidean distance between the coordinates of the first underwater vehicle's current position and the current target waypoint. g The Euclidean distance between the previous and current target waypoints is m2 = 0.2 in this embodiment. Condition (2) is used to assess whether the first underwater vehicle deviates significantly from the preset trajectory;

[0091] Condition (3): The current target waypoint is the mission endpoint, that is, the first underwater vehicle has reached the mission endpoint.

[0092] The method for obtaining "one first sample and one second sample corresponding to each second underwater vehicle" each time:

[0093] ①The status of the first underwater vehicle The input is fed into the first decision network to obtain the action. Actions will be communicated Transmit the status of each second underwater vehicle to each second underwater vehicle. and actions The inputs are respectively fed into the second decision network corresponding to the second underwater vehicle to obtain the actions of the second underwater vehicle.

[0094] ② Enable the first underwater vehicle to interact with the simulated sea area. During the interaction, calculate the actions performed by the first underwater vehicle based on its kinematic and dynamic models. The state after And based on the current state of the first underwater vehicle, the execution action is calculated according to the reward function of the first underwater vehicle. Reward value of the process State action state and reward value Transmit to the experience replay pool, and transfer the state action state and reward value As a first sample of the first underwater vehicle.

[0095] Each second underwater vehicle executes the actions output by its respective second decision network. The system interacts with the simulated sea area, and during the interaction, the actions performed by each second underwater vehicle are calculated based on its kinematic and dynamic models. The state after And based on the current state of the second underwater vehicle, the actions to be performed by the second underwater vehicle are calculated according to its reward function. Reward value of the process State action state Reward Value Transferred to the experience replay pool;

[0096] action For action The product of k and action The state of the first underwater vehicle The input is fed into the first decision network to obtain the action, and then the action is... Transferred to the experience replay pool;

[0097] The state in the experience replay pool action state Reward Value action and actions This serves as a second sample of the second underwater vehicle.

[0098] Example 3

[0099] In Example 2, after each voyage, the first underwater vehicle executes the "training method of underwater vehicle reinforcement learning model". The training method of underwater vehicle reinforcement learning model includes steps S1 and S2. Every 2T times the training method of underwater vehicle reinforcement learning model is executed, the updated first decision network and the updated second decision network are used to perform a voyage according to the voyage route in Example 2. If the first underwater vehicle and N second underwater vehicles can maintain formation and reach the mission endpoint in 5 consecutive voyages, the training is stopped.

[0100] Training methods for reinforcement learning models of underwater vehicles include:

[0101] S1. Randomly select M first samples from one voyage of the first underwater vehicle. Set up evaluation networks Q1, Q2, Q3, Q4, and decision network P1. Update the first decision network, evaluation network Q1, and evaluation network Q2 sequentially with the M first samples. The method for updating the first decision network, evaluation network Q1, and evaluation network Q2 sequentially for each first sample is as follows:

[0102] The state in the first sample The input is fed into the first decision network to obtain the actions of the first underwater vehicle. As an action The state in the first sample and actions The value is obtained by inputting the data into the evaluation network Q1. The state in the first sample and actions The value is obtained by inputting the data into the evaluation network Q2. The state in the first sample The input is fed into decision network P1 to obtain the actions of the first underwater vehicle. As an action And add normally distributed noise.

[0103] State and actions The value is estimated by inputting it into the evaluation network Q3. State and actions The value is estimated by inputting it into the evaluation network Q4. Selection of value estimation and valuation The smaller value in the middle is used as the value estimate Based on value estimation and the reward value in the sample The value estimate was calculated. (Fujimoto S, Van Hoof H, Meger D. Addressing Function Approximation Error in Actor-Critic Methods[Z].arXiv, 2018(2018).); Based on value estimation Value estimation Value estimation state and actions Calculate the loss function and update the first decision network, evaluation network Q1, and evaluation network Q2 of the first underwater vehicle.

[0104] The first underwater vehicle corresponds to four identical evaluation networks: Evaluation Network Q1, Evaluation Network Q2, Evaluation Network Q3, and Evaluation Network Q4. The input to each evaluation network is the state and actions of the first underwater vehicle; the output of each evaluation network is a value estimate, which is used to evaluate the actions input to the evaluation network to measure whether the actions output by the first decision network can enable the first underwater vehicle to navigate along a predetermined trajectory. Each evaluation network is constructed using a neural network and includes an input layer, a fourth hidden layer, a fifth hidden layer, and an output layer. The input layer has 12 nodes, the fourth hidden layer has 400 nodes, the fifth hidden layer has 300 nodes, and the output layer has 1 node. The activation functions of the fourth and fifth hidden layers are ReLU functions.

[0105] The decision network P1 has the same structure as the first decision network.

[0106] S2, randomly select M second samples from each second underwater vehicle during a single voyage. Each second underwater vehicle is configured with an evaluation network Q5, an evaluation network Q6, an evaluation network Q7, an evaluation network Q8, and a decision network P2. The M second samples are used to update the "N second decision networks and their corresponding evaluation networks Q5 and Q6" multiple times. The update method for each second decision network and its corresponding evaluation networks Q5 and Q6 is as follows:

[0107] The state of the second underwater vehicle in a single voyage during the second decision network's journey in a second sample. and actions The input is fed into the second decision network to obtain the action. As an action The states from one second sample of each of the N second underwater vehicles and actions The value estimate is obtained by inputting the data into the evaluation network Q5 corresponding to the second decision network. (The state of each of the second underwater vehicles in the second sample) and actions (Input together into the evaluation network Q5 corresponding to the same second underwater vehicle); the states from one second sample from each of the N second underwater vehicles and actions The value estimate is obtained by inputting the data into the evaluation network Q6 corresponding to the second decision network. (The state of each of the second underwater vehicles in the second sample) and actions (Input together into the same evaluation network Q6 corresponding to the second underwater vehicle); the state in the second sample corresponding to the second decision network. and actions The input is fed into decision network P2, corresponding to the second decision network, to obtain the actions of the second underwater vehicle. As an action And add normally distributed noise;

[0108] The states from one second sample of each of the N second underwater vehicles The individual actions of N second underwater vehicles The inputs are combined with the evaluation network Q7 corresponding to the second decision network to obtain the output value estimate. (The state of each of the second underwater vehicles in the second sample) and the actions of all second underwater vehicles The data is input together into the evaluation network Q7 corresponding to one second underwater vehicle; the states from one second sample of each of the N second underwater vehicles are then processed. and the actions of all second underwater vehicles The values ​​are then input together into the evaluation network Q8 corresponding to the second decision network to obtain the value estimate. (The states in the N second samples corresponding to all the second underwater vehicles) And the actions of N second underwater vehicles The data is input together into the evaluation network Q8 corresponding to a second underwater vehicle; value estimation is then selected. and valuation The smaller value in the middle is used as the value estimate

[0109] Based on value estimation and the reward value in the sample Calculate the target value estimate (Fujimoto S, Van Hoof H, Meger D. Addressing Function Approximation Error in Actor-Critic Methods[Z].arXiv, 2018(2018).); Based on value estimation Value estimation Target value estimation The state of the second sample of N second underwater vehicles And the actions of N second underwater vehicles Calculate the loss function and update the second decision network, evaluation network Q5, and evaluation network Q6 for each second underwater vehicle;

[0110] The four evaluation networks (evaluation network Q5, evaluation network Q6, evaluation network Q7, and evaluation network Q8) corresponding to one of the second underwater vehicles have the same structure. Each evaluation network of the second underwater vehicle is constructed using a neural network, including: an input layer, a sixth hidden layer, a seventh hidden layer, and an output layer. The number of nodes in the input layer is 12×N, the number of nodes in the sixth hidden layer is 400, the number of nodes in the seventh hidden layer is 300, and the number of nodes in the output layer is 1. The activation function of the sixth and seventh hidden layers is the ReLU function.

[0111] The decision network P2 has the same structure as the second decision network (each second underwater vehicle corresponds to one decision network P2).

[0112] The update method for the first decision network, evaluation network Q1, evaluation network Q2, second decision network, evaluation network Q5, and evaluation network Q6 is based on the MATD3 method: the first decision network is updated once for every n updates to evaluation network Q1 and evaluation network Q2; the second decision network is updated once for every n updates to evaluation network Q5 and evaluation network Q6.

[0113] The specific calculation formula of the loss function used in this embodiment can be found in the paper: Fujimoto S, VanHoof H, Meger D. Addressing Function Approximation Error in Actor-Critic Methods[Z].arXiv,2018(2018).

[0114] In this embodiment, T = 100, M = 100, and n = 2.

[0115] Example 4

[0116] Based on Example 3, an underwater vehicle reinforcement learning model further includes: an imitation learning compensator corresponding to each second underwater vehicle; when the second underwater vehicle cannot obtain the actions of the first underwater vehicle via communication, the imitation learning compensator is used to replace the first decision network; the input of the imitation learning compensator is the state. Imitate the output action of the learning compensator As an action The imitation learning compensator is constructed using a neural network and includes an input layer, an eighth hidden layer, a ninth hidden layer, a tenth hidden layer, an eleventh hidden layer, and an output layer. The input layer has 8 nodes, the eighth hidden layer has 1400 nodes, the ninth hidden layer has 1000 nodes, the tenth hidden layer has 600 nodes, the eleventh hidden layer has 300 nodes, and the output layer has 4 nodes. The activation function of the eighth, ninth, tenth, and eleventh hidden layers is the ReLU function, and the activation function of the output layer is the Tanh function.

[0117] Example 5

[0118] Based on Example 4, a training method for an underwater vehicle reinforcement learning model includes: during the repeated execution of S1 and S2 in Example 3, every T executions of S2 in Example 3, each imitation learning compensator is updated 1000 times; each time the imitation learning compensator is updated, 400 first samples are randomly selected from the experience replay pool, and the states in the first samples are... Input is fed into the imitation learning compensator, and the output is the action. Based on the actions in the first sample and the action output of the imitation learning compensator Calculate the loss function and update the imitation learning compensator; where the loss function of the imitation learning compensator is the action in the first sample. and the action output of the imitation learning compensator The mean square error between them.

[0119] Example 6

[0120] Based on Example 3, a deep reinforcement learning formation control method for underwater vehicles includes: deploying a trained underwater vehicle reinforcement learning model to the mission starting point in the marine environment; maintaining communication between the second underwater vehicle (two second underwater vehicles) and the first underwater vehicle (i.e., without imitation learning compensators); the first underwater vehicle continuously outputting the current decision action using its first decision network, calculating the state of the next decision based on kinematic and dynamic models, and navigating according to the navigation route based on the action; the second underwater vehicle acquiring the current decision action of the first underwater vehicle through communication, continuously outputting the next decision action using its second decision network, and calculating the state of the next decision based on kinematic and dynamic models, thereby achieving formation navigation with the first underwater vehicle.

[0121] Example 7

[0122] A deep reinforcement learning formation control method for underwater vehicles is basically the same as that in Example 6, except that the communication between the second underwater vehicle and the first underwater vehicle is limited. The method for obtaining the action of the first underwater vehicle is as follows: the second underwater vehicle obtains the state of the first underwater vehicle through communication, and inputs the state of the first underwater vehicle into the imitation learning compensator of each second underwater vehicle in Example 5 to obtain the action of the first underwater vehicle.

[0123] Example 8

[0124] A deep reinforcement learning formation control method for underwater vehicles is basically the same as that in Example 7, except that the communication between the second underwater vehicle and the first underwater vehicle is severely restricted. The method for obtaining the action of the first underwater vehicle is as follows: the state of the first underwater vehicle before executing the action of the current decision is calculated based on the action of the previous decision, the kinematic model and the dynamic model of the first underwater vehicle. The state is then input into the imitation learning compensator of each second underwater vehicle in Example 5 to obtain the action as the action of the current decision of the first underwater vehicle.

[0125] Example 9 (for comparison)

[0126] A deep reinforcement learning-based formation control method for underwater vehicles includes: using the deep reinforcement learning method MATD3 (Addressing Function Approximation Error in Actor-Critic Methods) to achieve formation control of two second underwater vehicles and a first underwater vehicle.

[0127] In Example 6, the second underwater vehicle acquires the actions of the first underwater vehicle. After increasing the input dimension of the decision network of the second underwater vehicle, its training complexity is basically unaffected, and it can still form a formation quickly and reliably.

[0128] The formation effects of Examples 9 and 7 are as follows: Figure 3 and Figure 4 As shown.

[0129] Figure 5 This is a comparison chart showing the convergence of the training method for the underwater vehicle reinforcement learning model of the present invention in Example 3 (PC-MATD3) and Example 9 (MATD3) during training. The chart shows that... Figure 5The middle curve represents the mean of the round-average reward of the second underwater vehicle (filled with 95% confidence intervals). In the early stage of training, the navigation paths of the second underwater vehicle and the first underwater vehicle in the deep reinforcement learning formation control method of the present invention are relatively short, and the round-average reward of the second underwater vehicle is higher. As the decision network of the first underwater vehicle is updated, its navigation path gradually becomes longer, the tracking error of the second underwater vehicle gradually increases, and its round-average reward decreases. As the decision network of the second underwater vehicle is updated, the tracking error of the second underwater vehicle gradually decreases, and its round-average reward recovers and converges. As can be seen from the figure, the round-average reward of the deep reinforcement learning formation control method of the present invention is smoother than that of the comparative method, and the variance fluctuation range of the shaded area is smaller, which is more superior.

[0130] The underwater vehicle deep reinforcement learning formation control method of the present invention (which still uses an imitation learning compensator to acquire the actions of the first underwater vehicle when communication is unrestricted) and Example 9, with mean distance error, standard deviation of distance error, mean angle error, and standard deviation of angle error in formation across multiple navigation routes, are shown in Table 1. As can be seen from Table 1, after the second underwater vehicle uses the imitation learning compensator to acquire the actions of the first underwater vehicle, the mean distance error and mean angle error of the formation are reduced by approximately 10%, and the standard deviation of distance error and standard deviation of angle error are reduced by more than 30%, demonstrating superior performance and stability.

[0131] Table 1

[0132]

[0133] The communication loss probabilities of Examples 8 and 9 were set to 0.2, 0.5, and 0.8, respectively. In Example 8, the second underwater vehicle used only the actions output by the imitation learning compensator as the actions of the first underwater vehicle for formation navigation. The mean distance error and mean angle error in a navigation route are shown in Table 2.

[0134] Among them, the communication loss probability is used to describe the proportion of the process of "calculating the position and state of the first underwater vehicle by using the action output of the first underwater vehicle in Example 8" in the overall formation navigation process.

[0135] Table 2

[0136]

[0137]

[0138] As shown in Table 2, as the probability of communication loss increases, the formation error of Example 9 gradually increases, and the anti-interference ability is poor. However, in Example 8, the imitation learning compensator is used to predict the intention of the first underwater vehicle and the position of the first underwater vehicle is estimated when communication is lost. The formation is almost unaffected by the communication loss. The underwater vehicle reinforcement learning model of the present invention continues to perform stably.

[0139] The present invention has been described above by way of example. It should be noted that any simple modifications, alterations or other equivalent substitutions that can be made by those skilled in the art without creative effort without departing from the core of the present invention fall within the protection scope of the present invention.

Claims

1. A reinforcement learning model for underwater vehicles, characterized in that, include: A first decision network and A second decision network, in which... Each first decision network corresponds to a first underwater vehicle, and each second decision network corresponds to a second underwater vehicle. The first underwater vehicle is used to navigate according to the route, and the second underwater vehicle is used to follow the first underwater vehicle while maintaining a fixed distance and angle. The first underwater vehicle and the second underwater vehicle each make a decision every Δt time interval. The input to the first decision network is the first underwater vehicle at the [missing information - likely a date or time]. The state of a decision The output of the first decision network is the action of the first underwater vehicle. ; The input to the second decision network is the second underwater vehicle corresponding to it in the [missing information - likely a date or time]. The state of a decision and actions , The constraint coefficients are used to determine the actions of the second underwater vehicle. ; The first underwater vehicle in The state of a decision , The coordinates of the current target waypoint of the first underwater vehicle in the world coordinate system. For the first underwater vehicle in the world coordinate system The coordinates of the location of each decision. Let be the expected yaw angle when the first underwater vehicle reaches the current target waypoint. This is the actual yaw angle when the first underwater vehicle reaches the current target waypoint. , , The coordinates of the first underwater vehicle are respectively: In its body coordinate system, along sequentially , , velocity in direction, The first underwater vehicle is located at coordinate point angular velocity at time , , The directions are, in order, the body coordinates of the first underwater vehicle. axis, shaft and Axial direction; action , , , The first underwater vehicle is positioned in the body coordinate system of the first underwater vehicle along... , , Acceleration in the direction of The angular acceleration of the first underwater vehicle; The second underwater vehicle was in The state of a decision , For the second underwater vehicle in the world coordinate system The coordinates of the location of each decision. , , These are the coordinates of the initial position of the second underwater vehicle relative to the initial position of the first underwater vehicle in the world coordinate system. , , The offset in direction. For the second underwater vehicle The actual yaw angle of the decision, , , The second underwater vehicle's first underwater vehicle in the body coordinate system of the second underwater vehicle Decision-making path , , velocity in direction, The angular velocity of the second underwater vehicle. , , The directions are, in order, the body coordinates of the second underwater vehicle. axis, shaft and Axial direction; The second underwater vehicle's actions , , , The second underwater vehicle is in the body coordinate system of the second underwater vehicle. , , Acceleration in the direction of The angular acceleration of the second underwater vehicle; Each second underwater vehicle corresponds to one imitation learning compensator, and the input of the imitation learning compensator is the state. Imitate the output action of the learning compensator As an action The imitation learning compensator is constructed using a neural network.

2. The underwater vehicle reinforcement learning model according to claim 1, characterized in that, The first decision network is constructed using a neural network, and the second decision network is constructed using a neural network.

3. The training method for the reinforcement learning model of an underwater vehicle as described in any one of claims 1 to 2, characterized in that, Each execution The training method for the reinforcement learning model of the underwater vehicle is to perform a voyage using an updated first decision network and an updated second decision network. If the first underwater vehicle and If both of Taiwan's second underwater vehicles can maintain formation and reach the mission endpoint, then training will cease. Each first sample is: state ,action ,state and reward value ,state Perform actions for the first underwater vehicle The reward value of the state after the (t+1)th decision For the first underwater vehicle to the first The reward value for each decision is calculated using a reward function: In the formula, For the first underwater vehicle in the +1 decision-related state of , For the first underwater vehicle in state The angle between the vector pointing from the current location's coordinates to the current target waypoint and its own velocity vector. and All are coefficients; Each second sample is: state ,action ,state Reward Value ,action and actions ,state Performing actions for the second underwater vehicle The state and action after the (t+1)th decision To input state into the first decision network The action obtained and The product of the two, the reward value For the second underwater vehicle to the first The reward value for each decision is calculated using a reward function: In the formula, and All are coefficients. , For the first Taiwan's second underwater vehicle is in status The distance between the current location's coordinates and its desired location's coordinates; For the first Taiwan's second underwater vehicle is in status The angle between the vector pointing from the current coordinate point to the desired position and its own velocity vector; The training methods for reinforcement learning models of underwater vehicles include S1 and S2: S1, randomly select one underwater vehicle during a single voyage. A first sample is used to set up an evaluation network. Evaluation Network Evaluation Network Evaluation Network and decision-making networks ,Will Each first sample sequentially updates the first decision network and the evaluation network. and evaluation network Each first sample sequentially updates the first decision network and the evaluation network. and evaluation network The method is as follows: The state in the first sample The input is fed into the first decision network to obtain the actions of the first underwater vehicle. As an action ; the state in the first sample and actions Input into the evaluation network To obtain a value estimate ; the state in the first sample and actions Input into the evaluation network ,get ; the state in the first sample Input into the decision network Obtain the action of the first underwater vehicle As an action ; State and actions Input into the evaluation network Obtain a value estimate ; will state and actions Input into the evaluation network Obtain a value estimate Select value estimation and valuation The smaller value in the middle is used as the value estimate According to value estimation and the reward value in the sample The value estimate was calculated. According to value estimation , Value estimation ,state and actions The loss function is calculated and the first decision network and evaluation network of the first underwater vehicle are updated. and evaluation network ; S2, randomly select one of the second underwater vehicles during a single voyage. For each second sample, an evaluation network is set up. A rating network A rating network A rating network and a decision network For "N second decision networks and their corresponding evaluation networks" and evaluation network "Updated multiple times, each second decision network and its corresponding evaluation network." and evaluation network The update method for each iteration is as follows: The state of the second underwater vehicle in a single voyage during the second decision network's journey in a second sample. and actions The input is fed into the second decision network to obtain the action. As an action ;Will The state of each of the second underwater vehicles in Taiwan in the second sample and actions The input is fed into the evaluation network corresponding to the second decision network. Value estimate obtained from ;Will The state of each of the second underwater vehicles in Taiwan in the second sample and actions The input is fed into the evaluation network corresponding to the second decision network. Value estimate obtained from ; The state in the second sample corresponding to the second decision network and actions Input to the decision network corresponding to the second decision network The actions of the second underwater vehicle were obtained. As an action ; Will The state of each of the second underwater vehicles in Taiwan in the second sample and The respective actions of Taiwan's second underwater vehicle Together, they are input into the evaluation network corresponding to the second decision network. Obtain the value estimate of the output ;Will The state of each of the second underwater vehicles in Taiwan in the second sample and the actions of all second underwater vehicles Together, they are input into the evaluation network corresponding to the second decision network. Obtain a value estimate Select value estimation and valuation The smaller value in the middle is used as the value estimate ; Based on value estimation and the reward value in the sample Calculate the target value estimate According to value estimation , Target value estimation , The state in the second sample of the second underwater vehicle of Taiwan and The movement of Taiwan's second underwater vehicle Calculate the loss function and update the second decision network and evaluation network for each second underwater vehicle. and evaluation network .

4. The training method according to claim 3, characterized in that, First decision-making network, evaluation network Evaluation Network Second decision network, evaluation network and evaluation network The update method follows the MATD3 approach: every update Secondary evaluation network and evaluation network If so, update the first decision network once; each update Secondary evaluation network and evaluation network If so, the second decision network is updated once.

5. The training method according to claim 4, characterized in that, The method for obtaining "one first sample and one second sample corresponding to each second underwater vehicle" each time: ①The status of the first underwater vehicle The input is fed into the first decision network to obtain the action. ; Actions will be communicated Transmit the status of each second underwater vehicle to each second underwater vehicle. and actions The inputs are respectively fed into the second decision network corresponding to the second underwater vehicle to obtain the actions of the second underwater vehicle. ; ② Enable the first underwater vehicle to interact with the simulated sea area. During the interaction, calculate the actions performed by the first underwater vehicle based on its kinematic and dynamic models. The state after And as the current state of the first underwater vehicle, the action to be executed is calculated based on the reward function of the first underwater vehicle. Reward value of the process , will state ,action ,state and reward value As a first sample of the first underwater vehicle; Each second underwater vehicle executes the actions output by its respective second decision network. The system interacts with the simulated sea area, and during the interaction, the actions performed by each second underwater vehicle are calculated based on its kinematic and dynamic models. The state after This is taken as the current state of the second underwater vehicle. Based on the reward function of the second underwater vehicle, the actions to be performed by the second underwater vehicle are calculated. Reward value of the process ; action For action and The product of actions The state of the first underwater vehicle The input is fed into the first decision network to obtain the action; State ,action ,state Reward Value ,action and actions This serves as a second sample of the second underwater vehicle.

6. The training method according to claim 5, characterized in that, Make the first underwater vehicle travel along the same route. Second-rate, The second underwater vehicle maintains the offset between itself and the first underwater vehicle and travels together, obtaining multiple first samples from the first underwater vehicle and multiple second samples from each second underwater vehicle. The number of first samples and second samples obtained in each voyage is multiple.

7. The training method according to claim 6, characterized in that, Each time "a first sample and a second sample corresponding to each second underwater vehicle" are obtained, the state in the first sample is... As the first underwater vehicle in its current state, the following judgments are made: Determine the Euclidean distance between the coordinates of the first underwater vehicle's current location and the current target waypoint. Is it less than the threshold? : If Euclidean distance Not less than the threshold If either condition (1) or condition (2) is not met, proceed to obtain "one first sample and one second sample corresponding to each second underwater vehicle"; if either condition (1) or condition (2) is met, the current voyage ends. If Euclidean distance Less than the threshold If either condition (2) or condition (3) is not met, the path point pointed to by the current target waypoint in the navigation route is selected as the next current target waypoint, and the process of obtaining "one first sample and one second sample corresponding to each second underwater vehicle" is repeated. If either condition (2) or condition (3) is met, the current voyage ends. Condition (1): The coordinates of the location of the first underwater vehicle in its current state are outside the simulated sea area; Condition (2): ,in, Let be the Euclidean distance between the coordinates of the first underwater vehicle's current location and the previous target waypoint. Let be the Euclidean distance between the coordinates of the first underwater vehicle's current location and the current target waypoint. This is the Euclidean distance between the previous waypoint and the current waypoint. Condition (3): The current target waypoint is the mission endpoint.

Images