An information-restricted underwater robot pursuit-evasion game motion strategy planning method based on deep reinforcement learning

Abstract

The application discloses a deep reinforcement learning-based information-restricted underwater robot pursuit-escape game motion strategy planning method, which comprises the following steps: constructing a depth two-dimensional underwater incomplete information game environment model; constructing an integrated graph attention mechanism deep reinforcement learning training framework for the pursuer, and designing a phased compound reward function; designing a regularized global planning and local obstacle avoidance method for the escapee to obtain an initial pursuit strategy, and training the escapee strategy by using a deep reinforcement learning method; implementing self-game evolution training, freezing and updating the network parameters of the pursuer and the escapee alternately, and obtaining a trained pursuit model; and performing multi-scene strategy verification and motion efficiency evaluation based on a simulation platform. The method combines the design of the compound reward function, effectively solves the search and capture problem under the information restriction, improves the flexibility of the escape strategy by using the self-game, improves the adaptability and generalization of the pursuit strategy, and has high task completion degree and environmental adaptability.

Description

Technical Field

[0001] This invention belongs to the field of underwater robot motion strategy planning technology, and specifically relates to a motion strategy planning method for information-constrained underwater robots in a pursuit-escape game based on deep reinforcement learning. Background Technology

[0002] With increasing global emphasis on marine resource development and maritime security, competition among nations in the marine domain is intensifying. Against this backdrop, Autonomous Underwater Vehicles (AUVs), as efficient, flexible, and safe tools for marine operations, have garnered significant attention due to their unique advantages. Compared to single AUVs, multi-AUV systems, with their spatial distribution, high mission efficiency, robustness, and flexibility, have become a key means of addressing complex marine missions. In underwater missions involving adversarial factors, the pursuit-escape game problem has become an important research direction for multi-AUV cooperative control. As a crucial area of ​​multi-agent research, pursuit-escape game theory is largely based on the assumption of complete information, meaning that both the pursuer and the escapee can obtain information about each other in real time. However, in real-world scenarios, only limited local observation information within a limited field of view is available. Therefore, in such scenarios, multi-AUV systems utilizing limited sensor information to complete pursuit tasks through cooperative movement have greater practical significance.

[0003] In common pursuit-escape game problems, opponents often employ fixed-rule strategies, resulting in pursuit strategies performing well against specific opponents. However, when the opponent's strategy changes, the strategies lack generalization ability and struggle to adapt to complex and dynamic environments. Therefore, achieving stable search and continuous pursuit under limited visibility conditions, while simultaneously improving the generalization ability of strategies under different adversarial intensities, has become a pressing technical problem to be solved.

[0004] Sun X, Sun B, Su Z. Cooperative pursuit-evasion game for multi-AUVs in the ocean current and obstacle environment[C]. International Conference on Intelligent Robotics and Applications. Singapore: Springer Nature Singapore, 2023: 201-213. This study applies a multi-step Q-learning algorithm to a pursuit-evasion game scenario for autonomous underwater vehicles (AUVs). A differential game model based on Markov Decision Process (MDP) is constructed to solve the multi-to-one pursuit-evasion problem in complex ocean environments. The introduction of a multi-step reward mechanism improves the convergence speed and stability of the algorithm. This research comprehensively considers the influence of irregular obstacles and random ocean currents on the game process. However, this game model assumes that the participants can obtain each other's instantaneous positional information, essentially belonging to a complete information game. There is a lack of research on incomplete information game problems under limited field of view, i.e., information-constrained conditions.

[0005] De Souza C, Newbury R, ​​Cosgun A, et al. Decentralized multi-agentpursuit using deep reinforcement learning[J]. IEEE Robotics and Automation Letters, 2021, 6(3): 4552-4559. This study employed the TD3 algorithm combined with shared experience to train a decentralized strategy, and introduced a course learning and formation scoring reward mechanism to improve capture efficiency. It is noteworthy that the escapee's strategy in this study is based on rule-based fixed patterns (such as fixed paths or simple potential field repulsion algorithms). This allows the pursuer to develop highly targeted emergent behaviors (such as "flanking" and "splitting" strategies) during training to specifically address these pre-defined behaviors. However, the study has not yet explored the escapee's ability to learn and evolve. Summary of the Invention

[0006] To address the aforementioned technical problems, the purpose of this invention is to provide a method for planning motion strategies in a chase-and-escape game for information-constrained underwater robots based on deep reinforcement learning.

[0007] This invention provides a method for planning motion strategies in an information-constrained underwater robot pursuit-escape game based on deep reinforcement learning, comprising:

[0008] Step 1: Construct a fixed-depth two-dimensional underwater incomplete information game environment model and perform initialization settings;

[0009] Step 2: For the pursuer, construct an integrated graph attention mechanism deep reinforcement learning training framework, design a phased composite reward function for incomplete information environment, and automatically switch the composite reward function according to the game stage to guide the strategy to maintain efficient exploration and pursuit even when the observation information is discontinuous;

[0010] Step 3: Design a rule-based global planning and local obstacle avoidance method for the escapee to obtain an initial pursuit strategy. Use deep reinforcement learning to train the escapee's strategy and improve the escapee's escape strategy.

[0011] Step 4: Implement self-game evolution training. By alternately freezing and updating the network parameters of the pursuer and the escapee, the strategy is iterated under the continuously escalating adversarial intensity. By utilizing game evolution under incomplete information conditions, the intelligence level and environmental adaptability of the pursuit strategy are enhanced, and a well-trained pursuit model is obtained.

[0012] Step 5: Perform multi-scenario strategy verification and motion performance evaluation based on the simulation platform. Input the pursuer's real-time restricted observation data, generate motion instructions through actor network inference, and execute strategy verification in multiple test scenarios with different obstacle distributions and initial game situations.

[0013] The present invention provides a motion strategy planning method for information-constrained underwater robots in a pursuit-escape game based on deep reinforcement learning, which has the following beneficial effects:

[0014] (1) The method of this invention, under the framework of multi-agent deep reinforcement learning, designs a phased composite reward function for incomplete information game scenarios with limited field of vision. The reward function of "search-pursuit-recapture" in three stages solves the problem of the target disappearing from the field of vision. The stage transition is automatically triggered according to whether the target is detected, so that the underwater robot can maintain a reasonable behavior strategy when the target is temporarily out of the field of vision, avoid strategy degradation or blind movement, and thus improve the probability of target recapture. At the same time, a dual-path graph attention neural network (GAT-O and GAT-T) is added to the evaluation network to obtain the spatial features between the underwater robot and obstacles. The adjacency relationship between multiple agents is expressed through graph structure, and the weights are dynamically allocated by attention mechanism to enhance the network's ability to express key spatial relationships, improve obstacle avoidance safety and collaborative capture efficiency, and thus improve the stability of the overall pursuit task.

[0015] (2) This invention introduces a learnable escape strategy and a self-game evolutionary training mechanism, enabling the escaper's strategy to be continuously adjusted and optimized during the training process, thereby forming a dynamic adversarial environment. This training method avoids the problem of overfitting to opponents with fixed rules and improves the generalization ability and environmental adaptability of the pursuit strategy under different obstacle distributions and different initial game situations. Through multi-scenario simulation verification, the method of this invention can maintain a high task completion rate and a low collision risk in complex environments. Attached Figure Description

[0016] Figure 1 This is a flowchart of a motion strategy planning method for an information-constrained underwater robot pursuit and escape game based on deep reinforcement learning, according to the present invention. Detailed Implementation

[0017] This invention provides a motion strategy planning method for information-constrained underwater robots in a pursuit-escape game based on deep reinforcement learning. Based on deep reinforcement learning algorithms, combined with multi-agent systems, graph attention neural networks, and a self-game framework, it addresses the problem of incomplete information games with limited field of view in complex marine environments. By employing self-game theory, it improves the flexibility of the escape strategy and enhances the adaptability and generalization of the pursuit strategy, resulting in high task completion rate and environmental adaptability.

[0018] like Figure 1 As shown, the present invention provides a method for planning motion strategies in a chase-and-escape game for information-constrained underwater robots based on deep reinforcement learning, comprising:

[0019] Step 1: Construct a fixed-depth two-dimensional underwater incomplete information game environment model and perform initialization settings, specifically:

[0020] Step 1.1: Simulate the actual environment by performing a fixed-depth two-dimensional model of the marine environment.

[0021] Step 1.2: Initialize the settings for the pursuer, the escapee, and the obstacles, including their position, speed, heading angle, and random seed parameters.

[0022] Step 1.3: Set the influence radius for obstacles in the environment and underwater robots, and set the corresponding radius and angle for the detection range and capture range of underwater robots, thereby establishing a constrained observation model.

[0023] Step 2: For the pursuer, construct an ensemble graph attention mechanism deep reinforcement learning training framework, design a phased composite reward function for incomplete information environments, and automatically switch the composite reward function according to the game stage to guide the strategy to maintain efficient exploration and pursuit even when the observed information is discontinuous. Specifically:

[0024] Step 2.1: Select the MAPPO algorithm from deep reinforcement learning algorithms, and design the observation space and action space according to the task characteristics; the observation space of the i-th underwater robot being chased is represented as follows:

[0025]

[0026] The observation space of the pursuer system at time t is ,in This represents the state of the i-th pursuing underwater robot. Let these represent the position, heading angle, and linear velocity of the i-th pursuing underwater robot, respectively. It represents the ratio of the actual return length of the rays emitted by the sensor in each direction to the maximum detection length, and is used to characterize the relative distribution of obstacles; This represents the actual length of the ray emitted along the corresponding detection direction after being blocked by an obstacle. k=1,2,…,5 represent the forward, left and right 30° and left and right 60° directions centered on the current heading angle of the underwater robot, respectively. Indicates the maximum detection length of the ray; , This represents the relative position of the i-th chasing underwater robot and its teammate. This represents the relative distance between the i-th pursuing underwater robot and its teammate. This represents the relative yaw angle between the i-th pursuing underwater robot and its teammate. This represents the azimuth angle of the i-th pursuing underwater robot and its teammates. ; This represents the relative position between the i-th pursuing underwater robot and the target escapee. This represents the relative distance between the i-th pursuing underwater robot and the target escapee. This represents the relative azimuth angle between the i-th pursuing underwater robot and the target escapee.

[0027] In the simulation environment, the motion space of the underwater robot is defined as the set of linear acceleration and angular velocity, as follows:

[0028]

[0029] The action space of the Chaser system at time t is , Let x and y represent the linear acceleration and angular velocity of the i-th pursuing underwater robot, respectively.

[0030] Step 2.2: Design multi-stage reward functions based on the process of the pursuit and escape game, and design corresponding reward functions for the search, pursuit and re-search stages.

[0031] A search reward was designed for the initial stage when the location of the escaped target is unknown. The search reward is calculated based on the change in the detection coverage area at the current moment compared to the coverage area at the previous moment. The reward function is as follows:

[0032]

[0033] in, It is the union of the areas covered by the detection range of all pursuers. It is the union of the areas covered by the detection range of all pursuers at the previous moment. It is the first proportionality constant, with a value between 2 and 3. Rewards are given for changes in the visible field of view.

[0034]

[0035] in, It represents the area newly explored by all pursuers at the current moment. It is the second proportionality constant, with a value ranging from 0.1 to 0.3. This represents a reward for exploring new areas.

[0036] The total reward for the search phase is as follows:

[0037]

[0038] When the pursuer detects the escapee and initiates a chase, the reward function for the chase phase is designed based on the change in distance between the pursuer and the escapee, the relationship between their headings and angles, and their speed ratios, as follows:

[0039]

[0040] in, It is the distance between the pursuer and the escapee. It is the distance between the pursuer and the escapee at the previous moment. It is the third proportionality constant, with a value between 1 and 2. This indicates the distance bonus during the pursuit phase.

[0041]

[0042] in, It is the pursuer's heading angle. It is the azimuth angle of the fleeing target relative to the pursuer. It is the fourth proportionality constant, with a value ranging from 0.1 to 0.2. This indicates the heading angle bonus during the pursuit phase.

[0043]

[0044] in, Represents the speed of the pursuer. Represents the maximum speed of the pursuer. It is the fifth proportionality constant, with a value ranging from 0.4 to 0.8. This indicates the speed bonus during the pursuit phase.

[0045] The total rewards for the pursuit phase are as follows:

[0046]

[0047] For the stage where the pursuer loses the target during the pursuit phase, the same reward function as in the pursuit phase is used. The last observed position of the escapee is used as the estimated target position for reward calculation. At the same time, the position of the target escapee in the agent's observation space is also used as the last observed position of the target escapee.

[0048] If the pursuer reaches the last observed location of the escaped target but fails to detect the escapee, the search phase resumes. The reward for this re-search phase is calculated based on the ratio of the escapee's potential activity range to the current coverage area; the specific reward function is as follows:

[0049]

[0050] in, Indicates the area that the escapee might flee to; This indicates the time elapsed since the escapee disappeared from sight. Indicates the maximum speed of the escapee. It is the union of the areas covered by the detection range of all pursuers. It is the union of the areas covered by the detection range of all pursuers at the previous moment.

[0051] A linear penalty is applied to the collision penalty between the obstacle and the pursuer, with the specific reward function as follows:

[0052]

[0053] in, This represents the distance between the i-th pursuing underwater robot and the obstacle, as well as other pursuers. Indicates the first collision safety threshold. It is the first penalty proportional constant, with a value of 3 to 5.

[0054] Meanwhile, if the pursuer successfully captures the escapee, the mission is considered successful and a large reward will be given; if the escapee successfully reaches the target point or the pursuer fails to capture the escapee, the mission is considered failed and a large penalty will be given.

[0055] For the pursuers, the following task outcome rewards are designed:

[0056]

[0057] in, This indicates the reward for the Pursuer mission. This represents the sixth proportionality constant, with a value ranging from 10 to 20.

[0058] The final composite reward function for the pursuer is:

[0059]

[0060] in, Different reward functions are applied to different states during the pursuit process, especially in the search phase. During the pursuit phase During the re-search phase .

[0061] Step 2.3: Establish an obstacle graph attention subnetwork (GAT-O) to receive the state information of the pursuer and the state information of the K nearest obstacles to the pursuer. Extract spatial relationship features through graph attention mechanism to help evaluate the spatial positional relationship between the pursuer and the obstacles.

[0062] Step 2.4: Establish the opponent state graph attention subnetwork (GAT-T) to receive state information between the pursuer and the escapee, extract game relationship features through graph attention mechanism, and help evaluate the position and state relationship between the pursuer and the escapee.

[0063] Step 2.5: Train the simulated scene according to the designed deep reinforcement learning framework, initialize the actor network and critic network in the model, and initialize the trajectory cache to store the complete interaction trajectory data obtained by downsampling under the current policy.

[0064] Step 2.6: Based on the current local observations The corresponding motion probability distribution is output through the actor network and the motion is obtained by sampling. The underwater robot is pursued based on the actions it performs. Receive the corresponding reward And enter the observation space of the next moment. The underwater robot tracks and stores the trajectory data obtained from each interaction in a trajectory cache.

[0065] Step 2.7: After completing the trajectory sampling for the preset step size or one round, calculate the discounted reward and advantage function using the data in the trajectory cache. The discounted reward is expressed as:

[0066]

[0067] in, This represents the discounted reward at time t, where T represents the end time of the current round or the end time of the current sampling segment. Indicates the discount factor. This represents the instantaneous reward obtained at time t+l. This represents the cumulative number of steps taken since the current moment.

[0068] Based on the discounted return, the advantage function is expressed as:

[0069]

[0070] in, This represents the dominance function at time t. This indicates the commentator network's view on the state at time t. State value estimation, This represents the state information of the commentator network input at time t.

[0071] The objective function for PPO-based pruning is expressed as follows:

[0072]

[0073] in, The parameters representing the actor network, This indicates the expectation of the data at the sampling time. This represents the dominance function at time t. Indicates the cutting factor. This represents the clipping function, used to restrict the probability ratio to an interval. Inside, The ratio of the probabilities of the new and old strategies for corresponding actions at time t is expressed as:

[0074]

[0075] in, This indicates that the current actor network is observing Down Output Action The probability, This indicates the output action of the old strategy under the same observations before the update. The probability, This indicates the actor's network parameters before the update. This indicates the sampling action at time t. This represents a local observation at time t.

[0076] PPO trimming of the objective function The actor network is updated in multiple rounds of small batches, while the critic network parameters are updated using the mean squared error loss function. After optimizing the current strategy, the trajectory cache is cleared, and the next round of sampling and training begins. The mean squared error loss function is expressed as follows:

[0077]

[0078] in, Indicates the network parameters of the critics. This indicates the critics' network's opinion on the state. Output value estimate This represents the discounted return at time t.

[0079] Step 2.8: By repeating steps 2.6 to 2.7, the underwater robot continuously interacts and iterates with the simulated environment to generate training data, thereby training a converged pursuer model.

[0080] Step 3: For the escapee, design a rule-based global planning and local obstacle avoidance method to obtain an initial pursuit strategy. Then, use deep reinforcement learning to train the escapee's strategy and improve its escape strategy. Specifically:

[0081] Step 3.1: Based on the escape target point, use the global path planning algorithm to plan the optimal collision-free path. At the same time, in order to adapt to the dynamic environment, a replanning mechanism is added to recalculate the global path when a fixed simulation step size is set or the current path fails.

[0082] Step 3.2: Treating the pursuer as a dynamic obstacle, an improved artificial potential field method is introduced based on global path planning. This method provides attraction to waypoints, guiding the escapee along the planned path, while simultaneously applying repulsive forces to both the pursuer and static obstacles. Specifically:

[0083] (1) Gravitational function formula:

[0084]

[0085] in, It is the field strength of the gravitational field where the escapee is located. It is the gravitational coefficient, used to adjust the magnitude of the gravitational field; This represents the Euclidean distance between the escapee and the local target waypoint.

[0086] (2) Formula for repulsive field function:

[0087]

[0088] in, It is the field strength of the repulsive field in which the escapee is located. It is the repulsion coefficient. This refers to the range of the repulsive force field; the repulsive force will only be calculated once the field enters this range. This represents the distance between the escapee and the obstacle.

[0089] (3) The formula for the force acting on the escapee is:

[0090]

[0091]

[0092] in, The force field experienced by the escapee. Forces suffered by the escapee The gravitational pull on the escapee The repulsive force experienced by the escapee; within the repulsive field, the closer the escapee is to the obstacle, the stronger the repulsive field and the greater the repulsive force experienced.

[0093] Step 3.3: Generate control commands for the escapee based on the direction of the resultant force, providing a rule-based strategy for the escapee in a dynamic adversarial environment.

[0094] Step 3.4: Based on the rule-based strategy, further train and optimize the escape strategy through reinforcement learning to obtain a more flexible escaper strategy, specifically as follows:

[0095] The observation space for the escapee is designed as follows:

[0096]

[0097] in This indicates the status of the escaping underwater robot, consistent with the design of the pursuing underwater robot. , This indicates the relative position of the escapee and the j-th pursuer. This represents the relative distance between the escapee and the j-th pursuer. Indicates the azimuth angles of the escapee and the j-th pursuer; ; This indicates the relative position and distance between the escapee and the target point.

[0098] The motion space of the escaping underwater robot is linear acceleration and angular velocity, consistent with the design of the pursuing underwater robot; the reward function of the escaping underwater robot is designed as a composite reward, as follows:

[0099] The reward design for the target point is as follows:

[0100]

[0101] in, It is the distance between the escapee and the target point. It is the azimuth angle of the escapee relative to the target. It is the escapee's heading angle; This is the first escape proportionality constant, with values ​​ranging from 1 to 3. It is the second escape ratio constant, with a value ranging from 0.2 to 0.5.

[0102] The penalties for obstacles and pursuers are designed as follows:

[0103]

[0104] in, Indicates the distance between the escapee and obstacles / pursuers. Indicates the second collision safety threshold; It is the second penalty proportional constant, with a value of 5 to 10.

[0105] For the escapee, reaching the target point is considered a mission success and results in a substantial reward; being caught by the pursuers is considered a mission failure and results in a substantial penalty. The mission outcome reward is designed as follows:

[0106]

[0107] in, This indicates the reward for the escapee's mission outcome. This represents the third escape proportionality constant, with a value ranging from 10 to 20.

[0108] The final reward function for the escapee is as follows:

[0109]

[0110] The simulation scenario is trained using a pre-designed deep reinforcement learning framework. The previously trained model is loaded for the pursuer, and a more flexible escaper strategy is obtained through further training.

[0111] Step 4: Implement self-game evolutionary training. By alternately freezing and updating the network parameters of the pursuer and the escapee, the strategy is iterated under continuously escalating adversarial intensity. Utilizing game evolution under incomplete information conditions, the intelligence level and environmental adaptability of the pursuit strategy are enhanced, resulting in a well-trained pursuit model. Specifically:

[0112] Step 4.1: Use a regularized escape strategy as the opponent to train the pursuer and obtain a preliminary convergent pursuer model.

[0113] Step 4.2: Fix the pursuer model and perform reinforcement learning training on the escaper model to obtain a converged escaper model, which makes the escaper also have more flexible strategies.

[0114] Step 4.3: Load the escaper model and regularized strategy obtained from the previous pre-training into the simulation environment, so that the escaper can exhibit diverse motion strategies in the environment. Initialize the kinematic state of the underwater robot (such as coordinates, heading angle and linear velocity, etc.), and retrain the pursuer to obtain the final converged pursuer model.

[0115] Step 5: Conduct multi-scenario strategy verification and motion performance evaluation based on the simulation platform. Input the pursuer's real-time restricted observation data, generate motion instructions through actor network inference, and execute strategy verification in multiple test scenarios with different obstacle distributions and initial game situations. Specifically:

[0116] Step 5.1: Build an underwater robot motion simulation environment in the Python environment, establish a simulation model that conforms to the kinematic constraints of the underwater robot, including the position, velocity and heading angle of the underwater robot; at the same time, establish an environment model, generate different obstacle distribution structures and different target point initial positions according to preset conditions, and generate multiple test scenarios.

[0117] Step 5.2: During the simulation, the simulation environment generates the observation space for the underwater robot to be pursued, i.e., the restricted observation information, based on the preset sensor perception range. The restricted observation information includes the relative azimuth angle between the underwater robot and the target escapee, the relative distance between the underwater robot and the target escapee, the heading angle and velocity information of the underwater robot itself, and the relative position information of obstacles within the perception range. The restricted observation information is used as the input state of the deep reinforcement learning actor network model, and the actor network outputs the motion control command of the pursuer at the current moment through forward inference.

[0118] Step 5.3: Based on the motion control commands of the pursuer, the state of the underwater robot is updated using the kinematic model of the underwater robot to obtain the position coordinates, heading angle, and velocity state of the pursuing underwater robot at the next moment; at the same time, the state and environmental state information of the escapee are updated, and the restricted observation information is regenerated based on the updated states of the pursuer and the escapee; the updated restricted observation information is fed back to the deep reinforcement learning model for motion decision-making at the next moment, thus forming a closed-loop simulation decision-making process based on "observation-decision-execution-feedback".

[0119] This invention designs a multi-stage composite reward function for incomplete information games with limited field of vision, effectively resolving the contradiction between search and pursuit. Furthermore, a graph attention neural network is incorporated into the comment network, enhancing the network's extraction and understanding of spatial features between the underwater robot and obstacles, thus improving the success rate of pursuit tasks compared to MAPPO. The self-game framework allows the escape strategy to have a certain learning ability, making it more flexible and improving the generalization and adaptability of the pursuit strategy to the environment.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the ideas of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for planning motion strategies in a chase-and-escape game for an information-constrained underwater robot based on deep reinforcement learning, characterized in that, include: Step 1: Construct a fixed-depth two-dimensional underwater incomplete information game environment model and perform initialization settings; Step 2: For the pursuer, construct an integrated graph attention mechanism deep reinforcement learning training framework, design a phased composite reward function for incomplete information environment, and automatically switch the composite reward function according to the game stage to guide the strategy to maintain efficient exploration and pursuit even when the observation information is discontinuous; Step 3: Design a rule-based global planning and local obstacle avoidance method for the escapee to obtain an initial pursuit strategy. Use deep reinforcement learning to train the escapee's strategy and improve the escapee's escape strategy. Step 4: Implement self-game evolution training. By alternately freezing and updating the network parameters of the pursuer and the escapee, the strategy is iterated under the continuously escalating adversarial intensity. By utilizing game evolution under incomplete information conditions, the intelligence level and environmental adaptability of the pursuit strategy are enhanced, and a well-trained pursuit model is obtained. Step 5: Perform multi-scenario strategy verification and motion performance evaluation based on the simulation platform. Input the pursuer's real-time restricted observation data, generate motion instructions through actor network inference, and execute strategy verification in multiple test scenarios with different obstacle distributions and initial game situations.

2. The method for planning motion strategies in a chase-and-escape game for information-constrained underwater robots based on deep reinforcement learning as described in claim 1, characterized in that, Step 1 specifically involves: Step 1.1: Simulate the actual environment by performing a fixed-depth two-dimensional model of the marine environment; Step 1.2: Initialize the settings for the pursuer, the escapee, and the obstacles, including their positions, speeds, heading angles, and random seed parameters; Step 1.3: Set the influence radius for obstacles in the environment and underwater robots, and set the corresponding radius and angle for the detection range and capture range of underwater robots, thereby establishing a constrained observation model.

3. The method for planning motion strategies in a chase-and-escape game for information-constrained underwater robots based on deep reinforcement learning as described in claim 1, characterized in that, Step 2 specifically involves: Step 2.1: Select the MAPPO algorithm from deep reinforcement learning algorithms, and design the observation space and action space according to the task characteristics; the observation space of the i-th underwater robot being chased is represented as follows: The observation space of the pursuer system at time t is ,in This represents the state of the i-th pursuing underwater robot. Let these represent the position, heading angle, and linear velocity of the i-th pursuing underwater robot, respectively. It represents the ratio of the actual return length of the rays emitted by the sensor in each direction to the maximum detection length, and is used to characterize the relative distribution of obstacles; This represents the actual length of the ray emitted along the corresponding detection direction after being blocked by an obstacle. k=1,2,…,5 represent the forward, left and right 30° and left and right 60° directions centered on the current heading angle of the underwater robot, respectively. Indicates the maximum detection length of the ray; , This represents the relative position of the i-th chasing underwater robot and its teammate. This represents the relative distance between the i-th pursuing underwater robot and its teammate. This represents the relative yaw angle between the i-th pursuing underwater robot and its teammate. This represents the azimuth angle of the i-th pursuing underwater robot and its teammates. ; This represents the relative position between the i-th pursuing underwater robot and the target escapee. This represents the relative distance between the i-th pursuing underwater robot and the target escapee. This represents the relative azimuth angle between the i-th pursuing underwater robot and the target escapee; In the simulation environment, the motion space of the underwater robot is defined as the set of linear acceleration and angular velocity, as follows: The action space of the Chaser system at time t is , Let x and y represent the linear acceleration and angular velocity of the i-th pursuing underwater robot, respectively. Step 2.2: Design multi-stage reward functions based on the process of the pursuit and escape game, and design corresponding reward functions for the search, pursuit, and re-search stages; A search reward was designed for the initial stage when the location of the escaped target is unknown. The search reward is calculated based on the change in the detection coverage area at the current moment compared to the coverage area at the previous moment. The reward function is as follows: in, It is the union of the areas covered by the detection range of all pursuers. It is the union of the areas covered by the detection range of all pursuers at the previous moment. It is the first proportionality constant, with a value between 2 and 3. Rewards are given for changes in the visible field of view. in, It represents the area newly explored by all pursuers at the current moment. It is the second proportionality constant, with a value ranging from 0.1 to 0.

3. The reward for exploring new areas is indicated by the following: The total reward for the search phase is as follows: When the pursuer detects the escapee and initiates a chase, the reward function for the chase phase is designed based on the change in distance between the pursuer and the escapee, the relationship between their headings and angles, and their speed ratios, as follows: in, It is the distance between the pursuer and the escapee. It is the distance between the pursuer and the escapee at the previous moment. It is the third proportionality constant, with a value between 1 and 2. Indicates the distance bonus during the pursuit phase; in, It is the pursuer's heading angle. It is the azimuth angle of the fleeing target relative to the pursuer. It is the fourth proportionality constant, with a value ranging from 0.1 to 0.

2. This indicates the heading angle bonus during the pursuit phase. in, Represents the speed of the pursuer. Represents the maximum speed of the pursuer. It is the fifth proportionality constant, with a value ranging from 0.4 to 0.

8. This represents the speed bonus during the pursuit phase; therefore, the total bonus during the pursuit phase is as follows: For the stage where the pursuer loses the target during the pursuit phase, the same reward function as in the pursuit phase is used. The last observed position of the escapee is used as the estimated target position for reward calculation. At the same time, the position of the target escapee in the agent's observation space is also used as the last observed position of the target escapee. If the pursuer reaches the last observed location of the escaped target but fails to detect the escapee, the search phase resumes. The reward for this re-search phase is calculated based on the ratio of the escapee's potential activity range to the current coverage area; the specific reward function is as follows: in, Indicates the area that the escapee might flee to; This indicates the time elapsed since the escapee disappeared from sight. Indicates the maximum speed of the escapee. It is the union of the areas covered by the detection range of all pursuers. It is the union of the areas covered by the detection ranges of all pursuers at the previous moment; A linear penalty is applied to the collision penalty between the obstacle and the pursuer, with the specific reward function as follows: in, This represents the distance between the i-th pursuing underwater robot and the obstacle, as well as other pursuers. Indicates the first collision safety threshold. It is the first penalty proportionality constant, with a value of 3 to 5; For the pursuers, the following task outcome rewards are designed: in, This indicates the reward for the Pursuer mission. This represents the sixth proportionality constant, with a value ranging from 10 to 20; The final composite reward function for the pursuer is: in, Different reward functions are applied to different states during the pursuit process, especially in the search phase. During the pursuit phase During the re-search phase ; Step 2.3: Establish an obstacle graph attention subnetwork to receive the state information of the pursuer and the state information of the K nearest obstacles to the pursuer. Extract spatial relationship features through graph attention mechanism to help evaluate the spatial positional relationship between the pursuer and the obstacles. Step 2.4: Establish an opponent state graph attention subnetwork to receive state information between the pursuer and the escapee, and extract game relationship features through graph attention mechanism to help evaluate the position and state relationship between the pursuer and the escapee. Step 2.5: Train the simulated scene according to the designed deep reinforcement learning framework, initialize the actor network and critic network in the model, and initialize the trajectory cache to store the complete interaction trajectory data obtained by downsampling under the current policy; Step 2.6: Based on the current local observations The corresponding motion probability distribution is output through the actor network and the motion is obtained by sampling. The underwater robot is pursued based on the actions it performs. Receive the corresponding reward And enter the observation space of the next moment. The underwater robot stores the trajectory data obtained from each interaction in the trajectory cache. Step 2.7: After completing the trajectory sampling of the preset step size or one round, calculate the discount reward and advantage function using the data in the trajectory cache, perform multiple rounds of small batch updates on the actor network based on the PPO pruning objective function, and update the critic network parameters using the mean squared error loss function. After completing the current strategy optimization, clear the trajectory cache and enter the next round of sampling and training loop. Step 2.8: By repeating steps 2.6 to 2.7, the underwater robot continuously interacts and iterates with the simulated environment to generate training data, thereby training a converged pursuer model.

4. The method for planning motion strategies in a chase-and-escape game for information-constrained underwater robots based on deep reinforcement learning as described in claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Based on the escape target point, use the global path planning algorithm to plan the optimal collision-free path. At the same time, in order to adapt to the dynamic environment, a replanning mechanism is added to recalculate the global path when a fixed simulation step size is set or the current path fails. Step 3.2: Treat the pursuer as a dynamic obstacle. Based on the global path planning, introduce an improved artificial potential field method to provide attraction to waypoints, guide the escapee to move along the planned path, and at the same time apply repulsive force to the pursuer and static obstacles. Step 3.3: Generate control commands for the escapee based on the direction of the resultant force, providing a rule-based strategy for the escapee in a dynamic adversarial environment; Step 3.4: Based on the rule-based strategy, further train and optimize the escape strategy through reinforcement learning to obtain a more flexible escaper strategy.

5. The method for planning motion strategies in a chase-and-escape game based on deep reinforcement learning for information-constrained underwater robots according to claim 4, characterized in that, Step 3.2 specifically involves: (1) Gravitational function formula: in, It is the field strength of the gravitational field where the escapee is located. It is the gravitational coefficient, used to adjust the magnitude of the gravitational field; This represents the Euclidean distance between the escapee and the local target waypoint; (2) Formula for repulsive field function: in, It is the field strength of the repulsive field in which the escapee is located. It is the repulsion coefficient. This refers to the range of the repulsive force field; the repulsive force will only be calculated once the field enters this range. Represents the distance between the escapee and the obstacle; (3) The formula for the force acting on the escapee is: in, The force field experienced by the escapee. Forces suffered by the escapee The gravitational pull on the escapee The repulsive force experienced by the escapee; within the repulsive field, the closer the escapee is to the obstacle, the stronger the repulsive field and the greater the repulsive force experienced.

6. The method for planning motion strategies in a chase-and-escape game for information-constrained underwater robots based on deep reinforcement learning according to claim 4, characterized in that, Step 3.4 specifically involves: The observation space for the escapee is designed as follows: in This indicates the status of the escaping underwater robot, consistent with the design of the pursuing underwater robot. , This indicates the relative position of the escapee and the j-th pursuer. This represents the relative distance between the escapee and the j-th pursuer. Indicates the azimuth angles of the escapee and the j-th pursuer; ; This indicates the relative position and distance between the escapee and the target point; The motion space of the escaping underwater robot is linear acceleration and angular velocity, consistent with the design of the pursuing underwater robot; the reward function of the escaping underwater robot is designed as a composite reward, as follows: The reward design for the target point is as follows: in, It is the distance between the escapee and the target point. It is the azimuth angle of the escapee relative to the target. It is the escapee's heading angle; This is the first escape proportionality constant, with values ​​ranging from 1 to 3. It is the second escape ratio constant, with a value ranging from 0.2 to 0.5; The penalties for obstacles and pursuers are designed as follows: in, Indicates the distance between the escapee and obstacles / pursuers. Indicates the second collision safety threshold; It is the second penalty proportionality constant, with a value of 5 to 10; For those who escape, the following task outcome reward is designed: in, This indicates the reward for the escapee's mission outcome. This represents the third escape proportionality constant, with a value ranging from 10 to 20; The final reward function for the escapee is as follows: The simulation scenario is trained using a pre-designed deep reinforcement learning framework. The previously trained model is loaded for the pursuer, and a more flexible escaper strategy is obtained through further training.

7. The method for planning motion strategies in a chase-and-escape game for information-constrained underwater robots based on deep reinforcement learning as described in claim 1, characterized in that, Step 4 specifically involves: Step 4.1: Use a regularized escape strategy as the opponent to train the pursuer and obtain a preliminary convergent pursuer model; Step 4.2: Fix the pursuer model and perform reinforcement learning training on the escaper model to obtain a converged escaper model, so that the escaper also has a more flexible strategy. Step 4.3: Load the escaper model and regularized strategy obtained from the previous pre-training into the simulation environment, so that the escaper can exhibit diverse motion strategies in the environment, initialize the kinematic state of the underwater robot, and retrain the pursuer to obtain the final converged pursuer model.

8. The method for planning motion strategies in a chase-and-escape game based on deep reinforcement learning for information-constrained underwater robots according to claim 1, characterized in that, Step 5 specifically involves: Step 5.1: Build an underwater robot motion simulation environment in the Python environment, establish a simulation model that conforms to the kinematic constraints of the underwater robot, including the position, velocity and heading angle of the underwater robot; at the same time, establish an environment model, generate different obstacle distribution structures and different target point initial positions according to preset conditions, and generate multiple test scenarios; Step 5.2: During the simulation, the simulation environment generates the observation space for the underwater robot pursuit based on the preset sensor perception range, i.e., the restricted observation information. The restricted observation information includes the relative azimuth angle between the underwater robot pursuit and the target escapee, the relative distance between the underwater robot pursuit and the target escapee, the underwater robot pursuit's own heading angle and velocity information, and the relative position information of obstacles within the perception range. The restricted observation information is used as the input state of the deep reinforcement learning actor network model, and the actor network outputs the motion control command of the pursuer at the current moment through forward inference. Step 5.3: Based on the motion control commands of the pursuer, update the state of the underwater robot in conjunction with the kinematic model of the underwater robot to obtain the position coordinates, heading angle and velocity state of the pursuing underwater robot at the next moment; at the same time, update the state and environmental state information of the escapee, and regenerate the restricted observation information based on the updated states of the pursuer and the escapee. The updated constrained observation information is fed back to the deep reinforcement learning model for motion decision-making in the next time step, thus forming a closed-loop simulation decision-making process based on "observation-decision-execution-feedback".

Images