Multi-UUV formation control method and system based on reinforcement learning

Abstract

The invention discloses a multi-UUV formation control method and system based on reinforcement learning, and belongs to the technical field of multi-UUV cooperative formation control. Aiming at the task requirements of large-range detection and high-precision control and the problems that the formation of multiple UUVs is difficult to maintain and the formation stability is poor in an unknown environment, the method comprises the following steps: firstly, constructing a Markov decision model for a multi-UUV formation tracking process; the invention provides a multi-UUV (Unmanned Underwater Vehicle) dual-delay depth deterministic strategy gradient (MUTD3) reinforcement learning method based on an actuator-evaluator framework, and further provides a multi-UUV dual-delay depth deterministic strategy gradient (MUTD3) reinforcement learning method based on an actuator-evaluator framework. According to the method, collision between UUVs is avoided by designing a potential field function, and a multi-UUV reinforcement learning network is updated by adopting an experience playback mechanism, so that each U monomer is guided to follow a generated virtual target point, an expected formation is realized, and finally stable and coordinated multi-UUV formation control is completed.

Description

Technical Field

[0001] This invention relates to the field of multi-UUV cooperative formation control technology, and in particular to a multi-UUV formation control method and system based on reinforcement learning, which is applicable to scenarios such as large-scale marine environmental monitoring and underwater search and rescue. Background Technology

[0002] With the development of new materials and technologies, unmanned underwater vehicles (UUVs) play an important role in underwater target detection, search, and processing. Due to limitations in speed, range, and payload capacity, a single UUV cannot meet the requirements of large-scale detection and high-precision control in complex marine environments. Multi-UUV collaborative operations can effectively optimize underwater mission efficiency, and multi-UUV formation control is one of the key technologies. Summary of the Invention

[0003] The purpose of this invention is to address the challenges of maintaining formation and poor formation stability of multiple UUVs in unknown environments, in order to meet the requirements of large-scale detection and high-precision control tasks.

[0004] This invention proposes a multi-UUV formation control method and system based on reinforcement learning, the core of which includes the following technical solutions:

[0005] A multi-UUV formation control method and system based on reinforcement learning includes the following steps:

[0006] Step 1: Establish a Markov decision process model for the multi-UUV system based on the characteristics of the underactuated UUV dynamics model; set the expected planning path for each UUV; set the maximum number of steps per round; initialize the number of steps per round and the experience replay pool.

[0007] Step 2: Obtain the dynamic expected state point of each UUV from the expected planned path of each UUV; through environmental perception and combined with the forward-looking sonar detection data of each UUV, obtain the local observation information of each UUV, the distance between each UUV, the actual state point of each UUV, the obstacle distance matrix and the local target point; based on the local observation information of each UUV, obtain the observation value of each UUV.

[0008] Step 3: Combine the local target points of all UUVs to obtain the local target set points of the UUV formation; combine the actual state points of all UUVs to obtain the actual state set points of the UUV formation; combine the dynamic expected state points of all UUVs to obtain the dynamic expected state set points of the UUV formation; combine the observations of all UUVs to obtain the observation set of the UUV formation; calculate the reward function of the UUV formation based on the actual state set points of the UUV formation and the actual state points of each UUV.

[0009] Step 4: Input the actual state set point S of the UUV formation, the dynamic expected state set point S′ of the UUV formation, the observation set of the UUV formation, and the reward function of the UUV formation into the multi-UUV reinforcement learning network to obtain the action values ​​of the actual state of the UUV formation and the action values ​​of the expected state of the UUV formation. Combine the reward function of the UUV formation to perform external obstacle avoidance, and avoid collisions between UUVs by calculating the potential field function of each UUV. Determine whether the UUV formation control of this round is completed based on the number of steps in each round.

[0010] Furthermore, the reward function R for the UUV formation described in step 3 is the guiding penalty r. guide The final reward r for each UUV i goal The final reward set r goal and sonar penalty r sonar The accumulation of.

[0011] Furthermore, the guiding penalty r guide The calculation methods include:

[0012]

[0013] Where (x, y) are the coordinates of the current actual state assembly point S of the UUV formation; (x goal ,y goal S is the local target assembly point of the UUV formation. goal coordinates; w time For parameters affecting sailing time; w distance The current position of the UUV formation and the local target assembly point S goal The Euclidean distance influence coefficient; w course ψ is the heading influence coefficient; ψ is the set of heading angles of the UUV formation.

[0014] Furthermore, the final reward for each UUV The calculation methods include:

[0015]

[0016] Where f is a constant; S represents the local target point of the i-th UUV; i Let r be the current actual state point of the i-th UUV. Further, the sonar penalty r... sonar The calculation methods include:

[0017] r sonar = (δ-1)×D max

[0018] Where δ is the obstacle evaluation factor; Dmax This represents the maximum detection range.

[0019] Furthermore, step 4 specifically includes the following steps:

[0020] Step 4.1: Determine whether a collision occurs with an obstacle based on the reward function R. If a collision occurs, update the dynamic expected state point and number of steps per round for each UUV, and return to step 3.

[0021] If no collision occurs, proceed to the next step.

[0022] Step 4.2: Calculate the potential field function for each UUV and determine whether a collision occurs between UUVs; if a collision occurs, update the dynamic expected state point and the number of steps per round for the UUVs that collided based on the potential field function, and return to step 3;

[0023] If no collision occurs, proceed to the next step.

[0024] Step 4.3: Determine whether the number of steps in each round has reached the maximum number of steps in each round. If not, have each UUV drive to its corresponding dynamic desired state point, update the number of steps in each round, and return to step 2.

[0025] If the maximum number of steps is reached, exit and proceed to the next step;

[0026] Step 4.4: Store the relevant data of successfully avoiding obstacles and maintaining obstacle avoidance inside the UUV to reach the dynamic desired state point into the experience replay pool, update the relevant parameters of the multi-UUV reinforcement learning network according to the experience replay pool, and complete this UUV formation control process.

[0027] Furthermore, the method for calculating the potential field function includes:

[0028]

[0029] Where, k rep The repulsive force scale factor; Let x be the safety range of the i-th UUV; x and y are the major and minor axes of the base, respectively; k1 and k2 are both scaling factors; u i and v i Let X and Y be the longitudinal and lateral velocities of the i-th UUV, respectively; (X, Y) are the center coordinates of the j-th UUV. Let be the distance between the i-th UUV and the j-th UUV. Let 'a' represent the influence radius of the j-th UUV.

[0030] A computer device / apparatus / system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

[0031] A computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

[0032] A computer program product comprising a computer program / instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

[0033] The beneficial effects of this invention are as follows:

[0034] Compared with existing technologies, the multi-UUV formation control method and system based on reinforcement learning of this invention utilizes the dynamic modeling characteristics of underactuated UUVs to construct a Markov decision process model for multiple UUVs, providing a modeling foundation for subsequent reinforcement learning methods and systems. Furthermore, by designing the MUTD3 reinforcement learning method and system, the training efficiency and stability of the overall network are effectively improved. The PER policy is introduced into reinforcement learning, accelerating the convergence speed and performance improvement of the policy network through priority sampling of high-value experience samples. This invention also designs an artificial potential field method based on UUV characteristics to ensure the safety and formation stability among formation members. Attached Figure Description

[0035] Figure 1 This invention relates to a multi-UUV formation control method and system structure based on reinforcement learning.

[0036] Figure 2 This is a diagram illustrating the reinforcement learning method and system architecture based on MUTD3 of the present invention.

[0037] Figure 3 This is a two-dimensional planar hexagonal formation trajectory diagram of the present invention.

[0038] Figure 4 This is the obstacle environment hexagonal formation distance error diagram of the present invention.

[0039] Figure 5 The obstacle environment hexagonal formation heading error diagram of the present invention Detailed Implementation

[0040] The following is in conjunction with the appendix Figure 1 and Figure 2 The present invention will be further described below.

[0041] A multi-UUV formation control method and system based on reinforcement learning includes the following steps:

[0042] refer to Figure 1Step 1: Establish a Markov decision process model for the multi-UUV system based on the characteristics of the underactuated UUV dynamics model; set the expected planning path for each UUV; set the maximum number of steps per round; initialize the number of steps per round and the experience replay pool.

[0043] Step 2: Obtain the dynamic expected state point of each UUV from the expected planned path of each UUV; through environmental perception and combined with the forward-looking sonar detection data of each UUV, obtain the local observation information of each UUV, the distance between each UUV, the actual state point of each UUV, the obstacle distance matrix and the local target point; based on the local observation information of each UUV, obtain the observation value of each UUV.

[0044] Step 3: Combine the local target points of all UUVs to obtain the local target set points of the UUV formation; combine the actual state points of all UUVs to obtain the actual state set points of the UUV formation; combine the dynamic expected state points of all UUVs to obtain the dynamic expected state set points of the UUV formation; combine the observations of all UUVs to obtain the observation set of the UUV formation; calculate the reward function of the UUV formation based on the actual state set points of the UUV formation and the actual state points of each UUV.

[0045] The reward function R of the UUV formation is the guiding penalty r. guide The final reward for each UUV The final reward set r goal and sonar penalty r sonar The accumulation of.

[0046] The guidance penalty r guide satisfy:

[0047]

[0048] Where (x, y) are the coordinates of the current actual state assembly point S of the UUV formation; (x goal ,y goal S is the local target assembly point of the UUV formation. goal coordinates; w time For parameters affecting sailing time; w distance The current position of the UUV formation and the local target assembly point S goal The Euclidean distance influence coefficient; w course ψ is the heading influence coefficient; ψ is the set of heading angles of the UUV formation.

[0049] The final reward for each UUV satisfy:

[0050]

[0051] Where f is a constant; S represents the local target point of the i-th UUV; i Let be the current actual state point of the i-th UUV.

[0052] The sonar penalty satisfy:

[0053] r sonar = (δ-1)×D max

[0054] Where δ is the obstacle evaluation factor; D max This represents the maximum detection range.

[0055] refer to Figure 2 Step 4: Input the actual state set point S of the UUV formation, the dynamic expected state set point S′ of the UUV formation, the observation set of the UUV formation, and the reward function of the UUV formation into the multi-UUV reinforcement learning network to obtain the action values ​​of the actual state of the UUV formation and the action values ​​of the expected state of the UUV formation. Combine the reward function of the UUV formation to perform external obstacle avoidance, and avoid collisions between UUVs by calculating the potential field function of each UUV. Determine whether the UUV formation control of this round is completed based on the number of steps in each round.

[0056] Step 4.1: Determine whether a collision occurs with an obstacle based on the reward function R. If a collision occurs, update the dynamic expected state point and number of steps per round for each UUV, and return to step 3.

[0057] If no collision occurs, proceed to the next step.

[0058] Step 4.2: Calculate the potential field function of each UUV and determine whether a collision occurs between each UUV; if a collision occurs, update the dynamic expected state point and the number of steps per round of the UUV that collided according to the potential field function, and return to step 3.

[0059] If no collision occurs, proceed to the next step.

[0060] The method for calculating the potential field function includes:

[0061]

[0062] Where, k rep The repulsive force scale factor; Let x be the safety range of the i-th UUV; x and y are the major and minor axes of the base, respectively; k1 and k2 are both scaling factors; u i and v i Let X and Y be the longitudinal and lateral velocities of the i-th UUV, respectively; (X, Y) are the center coordinates of the j-th UUV. Let be the distance between the i-th UUV and the j-th UUV. Let 'a' represent the influence radius of the j-th UUV.

[0063] Step 4.3: Determine whether the number of steps in each round has reached the maximum number of steps in each round. If not, have each UUV drive to its corresponding dynamic desired state point, update the number of steps in each round, and return to step 2.

[0064] If the maximum number of steps is reached, exit and proceed to the next step.

[0065] Step 4.4: Store the relevant data of successfully avoiding obstacles and maintaining obstacle avoidance inside the UUV to reach the dynamic desired state point into the experience replay pool, update the relevant parameters of the multi-UUV reinforcement learning network according to the experience replay pool, and complete this UUV formation control process.

[0066] Example

[0067] This invention is based on the Windows 10 system, and the relevant experiments were carried out in the Python 3.7 editor. The software environment parameter settings are shown in Table 1.

[0068] Table 1 Experimental Simulation Environment Parameters

[0069] Bag Bag OpenAIGym 0.26.2 Numpy 1.21.6 Stable-Baselines3 2.0.0 Cloudpickle 2.2.1 PyTorch 1.9.1+cu111 Gymnasium 0.28.1

[0070] Additionally, the discount factor was set to 0.95, the learning rate to 7e-4, the soft update parameter to 0.005, the experience pool capacity to 2e5, the batch size to 1024, the policy network update frequency to 2, the total number of training steps to 1e7, the maximum number of steps per round to 1000, and the simulation time interval to 0.5s.

[0071] The virtual environment was set up as follows: a 500m × 500m two-dimensional area was set up, and the initial data and expected endpoint data of the 6 UUVs were shown in Table 2. Three obstacles were randomly generated, one of which was a circular obstacle with a center coordinate of (98, 177) and a radius of 8, and the other two were circular obstacles with a center coordinate of (250, 143) and (205, 255) and a radius of 6.

[0072] Experimental simulation results are as follows Figure 3 , Figure 4 , Figure 5 As shown in the three images, the six UUVs, triggered from their initial positions, successfully moved towards the target location in a hexagonal formation. Clearly, this method and system can effectively track formation trajectories. From the obstacles in the images, it can be seen that UUV2 and UUV3 noticeably deflected when approaching the circular obstacle above, indicating that they actively took obstacle avoidance maneuvers. The initial positions of UUV1, UUV5, and UUV6 were close together. Figure 5It can be seen that it made a clear course deviation, demonstrating good internal obstacle avoidance and formation maintenance capabilities. This is due to the effective performance of the internal obstacle avoidance method and system based on artificial potential fields.

[0073] Table 2. Parameters related to initial and desired positions

[0074] UUV model UUV initial position initial velocity u Initial heading ψ Desired location Expected speed u Desired heading ψ 1 (115,110) 0 0 (409.4,195.4) 0.5 π / 4 2 (110,120) 0 0 (417.5,189.5) 0.5 π / 4 3 (130,110) 0 0 (426.6,193.7) 0.5 π / 4 4 (105,125) 0 0 (427.6,203.6) 0.5 π / 4 5 (110,110) 0 0 (419.4,209.4) 0.5 π / 4 6 (100,100) 0 0 (410.3,205.3) 0.5 π / 4

[0075] Figure 4 The diagram shows how the distance error of six UUVs in a hexagonal formation in an environment with unknown obstacles changes with the number of steps. In the initial stage, the errors of each UUV are relatively large, but they all converge to 0 after about 400 steps, indicating that the method and system of this invention effectively avoids obstacles and makes adjustments in the early stage, enabling each UUV to return to the ideal formation structure.

[0076] Figure 5 This shows the variation of heading error as a number of steps when six UUVs form a hexagonal formation in an environment with unknown obstacles. It is clearly observed that the heading error experiences three fluctuations between 0 and 500 steps. Combined with... Figure 3 The trajectory analysis shows that these three error deviations correspond to the following key stages: 1. Each UUV starts from its initial position and heads towards the target trajectory. 2. During formation, some UUVs approach obstacles and perform effective obstacle avoidance maneuvers. 3. The UUV formation route ends at the end of the straight path and begins to enter the turning section, during which path adjustment operations are performed. Therefore, the fluctuations in these stages are reasonable. After approximately 500 steps, all deviations converge to zero, indicating that the system has completed obstacle avoidance and path adjustment, successfully restoring and maintaining the hexagonal formation structure.

[0077] The experimental results demonstrate that the formation trajectory tracking method proposed in this invention exhibits excellent obstacle avoidance capability, formation stability, and error convergence performance in marine environments facing unknown obstacles. Therefore, this method and system have significant practical implications for real-world multi-UUV formation applications.

[0078] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-UUV formation control method based on reinforcement learning, characterized in that The method comprises the following steps: Step 1: establishing a Markov decision process model of the multi-UUV system based on the characteristics of the under-actuated UUV dynamics model; setting a desired planning path for each UUV; setting a maximum number of steps per round; initializing the number of steps per round and the experience replay pool; Step 2: obtaining a dynamic desired state point of each UUV from the desired planning path of each UUV; obtaining local observation information of each UUV, distance between each UUV, actual state point of each UUV, obstacle distance matrix and local target point by combining the front-looking sonar detection data of each UUV with the environmental perception; obtaining an observation value of each UUV according to the local observation information of each UUV; Step 3: combining the local target points of all UUVs to obtain a local target set point of the UUV formation; combining the actual state points of all UUVs to obtain an actual state set point of the UUV formation; combining the dynamic desired state points of all UUVs to obtain a dynamic desired state set point of the UUV formation; combining the observation values of all UUVs into an observation value set of the UUV formation; calculating a reward function of the UUV formation according to the actual state set point of the UUV formation and the actual state point of each UUV; Step 4: inputting the actual state set point S of the UUV formation, the dynamic desired state set point S' of the UUV formation, the observation value set of the UUV formation and the reward function of the UUV formation into the multi-UUV reinforcement learning network to obtain an action value of the actual state of the UUV formation and an expected state action value of the UUV formation, combining the reward function of the UUV formation to perform external obstacle avoidance, and avoiding collision between UUVs by calculating a potential field function of each UUV; determining whether the UUV formation control in this round is completed according to the number of steps per round.

2. The multi-UUV formation control method based on reinforcement learning according to claim 1, wherein: The reward function R of the UUV formation described in step 3 is the sum of the guidance reward r guide , the final reward r i goal of each UUV, the final reward set r goal composed of the sum of the sonar reward r sonar and the sonar penalty r 3. The multi-UUV formation control method based on reinforcement learning according to claim 2, wherein, The guiding penalty r guide The calculation method comprises: wherein (x, y) is the coordinate of the current actual set point S of the UUV formation; (x goal ,y goal ) is the coordinate of the local target set point S goal of the UUV formation; w time is the sailing time influence parameter; w distance is the Euclidean distance influence coefficient of the current position of the UUV formation and the local target set point S goal ; w course is the heading influence coefficient; and ψ is the heading angle set of the UUV formation.

4. The multi-UUV formation control method based on reinforcement learning of claim 2, wherein, the final reward r of each UUV i goal The calculation method comprises: where f is a constant; is a local target point of the i-th UUV; S i is a current actual state point of the i-th UUV.

5. The multi-UUV formation control method based on reinforcement learning of claim 2, wherein, The sonar penalty r sonar The method of calculating comprises: r sonar = (δ - 1) x D max Wherein, δ is the obstacle evaluation factor; D max is the maximum detection distance.

6. The multi-UUV formation control method based on reinforcement learning of claim 2, wherein, Step 4 specifically comprises the following steps: Step 4.1: determining whether a collision with an obstacle occurs according to the reward function R, if a collision occurs, updating the dynamic desired state point of each UUV and the number of steps per round, and returning to step 3; if no collision occurs, the next step is executed; Step 4.2: calculating the potential field function of each UUV to determine whether a collision occurs between each UUV; if a collision occurs, updating the dynamic desired state point of the UUV that collides and the number of steps per round according to the potential field function, and returning to step 3; if no collision occurs, the next step is executed; Step 4.3: determining whether the number of steps per round reaches the maximum number of steps per round, if not, causing each UUV to move towards the dynamic desired state point corresponding to the UUV, updating the number of steps per round, and returning to step 2; if the maximum number of steps is reached, the next step is executed; Step 4.4: storing the relevant data of successfully avoiding obstacles and maintaining UUV internal obstacle avoidance to reach the dynamic desired state point into the experience replay pool, updating the relevant parameters of the multi-UUV reinforcement learning network according to the experience replay pool, and completing the UUV formation control process this time.

7. The multi-UUV formation control method based on reinforcement learning according to claim 6, wherein, The calculation method of the potential field function comprises: where k rep is the repulsion scale factor; is the safety range of the ith UUV; x and y are the base major and minor semi-axes, respectively; k1 and k2 are both proportional coefficients; u i and v i are the longitudinal and lateral velocities of the ith UUV, respectively; (X, Y) is the center coordinate of the jth UUV; is the distance between the ith UUV and the jth UUV. a represents the influence radius of the jth UUV.

8. A computer apparatus / device / system comprising a memory, a processor, and a computer program stored on the memory, characterized in that: The processor executes the computer program to realize the steps of the method in any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that: The computer program / instructions are executed by the processor to realize the steps of the method in any one of claims 1 to 7.

10. A computer program product comprising computer programs / instructions, characterized in that: The computer program / instructions, when executed by the processor, implement the steps of the method of any one of claims 1 to 7.

Images