Unmanned ship hunting confrontation decision-making method based on adaptive course reinforcement learning

Abstract

The invention discloses an unmanned ship hunting confrontation decision-making method based on multi-agent course reinforcement learning. The method comprises the following steps: constructing a marine confrontation simulation environment comprising a dynamic target and multiple reefs; designing a normalized state space to represent a friend or foe movement situation, a multi-scale reward function and a continuous action space; a self-adaptive course scheduler is deployed in the centralized training decentralized execution framework, and the training environment complexity, the action noise intensity and the strategy entropy coefficient are dynamically adjusted; and training the decision model by adopting a multi-agent near-end strategy optimization (MAPPO) algorithm. According to the scheduler, the environmental difficulty is dynamically upgraded and degraded based on the sliding window success rate, the training round number and the performance stagnation counter, and the problems of sparse reward and strategy generalization are solved. Experiments show that the method significantly improves the hunting success rate, shortens the task time and reduces the collision rate.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent unmanned system control, and in particular relates to an unmanned boat encirclement and capture confrontation decision-making method based on adaptive curriculum reinforcement learning. Background Art

[0002] In recent years, in localized wars such as confrontations and conflicts, intelligent unmanned vehicles, such as drones, have demonstrated significant combat effectiveness. In particular, suicide drones, with their stealth and high maneuverability, have brought unprecedented challenges to traditional defense systems and combat rules. Both sides have widely deployed suicide drones, carrying out diverse missions such as precision strikes on high-value targets, disabling key nodes, deceiving and depleting target firepower, and disrupting decision-making and deployment.

[0003] With the rapid development of land and air unmanned swarm game confrontation technology, research on the game confrontation of marine unmanned boat swarms has also been gradually carried out, aiming to promote the construction of an integrated land, sea and air intelligent unmanned swarm collaborative combat system. However, the current marine unmanned boat swarm confrontation still faces the following technical bottlenecks:

[0004] (1) Simplification of scenarios: Existing simulation environments are difficult to simulate the complexity of real ocean confrontations;

[0005] (2) Low training efficiency: In a confrontation scenario where both the enemy and the friendly party are learning agents, traditional methods converge slowly;

[0006] (3) Reward sparsity: Positive reward signals are scarce in the roundup task, which makes strategy learning difficult;

[0007] (4) Poor generalization: The trained decision-making model is difficult to adapt to the dynamically changing adversarial environment.

[0008] At the same time, although multi-agent reinforcement learning provides a new paradigm for solving the collaborative decision-making of multiple unmanned boats, it still faces severe challenges in efficiency and generalization in the above-mentioned complex confrontation scenarios. Summary of the Invention

[0009] The purpose of this invention is to propose an unmanned boat capture confrontation decision-making method based on adaptive curriculum reinforcement learning to address the problems of scene simplification in unmanned boat capture confrontation tasks, low training and learning efficiency in complex confrontation scenarios where both the enemy and the friendly forces are learning intelligent agents, sparse rewards and poor generalization.

[0010] To achieve the above objectives, the present invention provides a technical solution: an unmanned boat encirclement and capture confrontation decision-making method based on adaptive curriculum reinforcement learning, the specific steps of which are as follows:

[0011] S1: Construct a two-dimensional ocean confrontation simulation scenario including dynamic targets, obstacles, and friendly islands and reefs, set the conditions for successful capture and mission termination, and establish a kinematic model for the unmanned boat;

[0012] S2: Design the normalized state space and the continuous action space based on the expected speed and angular velocity;

[0013] S3: Design a multi-scale reward function that includes distance reward, angle coordination reward, speed reward, collision penalty and task completion reward;

[0014] S4: Model the multi-UAV capture confrontation as a Markov game problem;

[0015] S5: Build a multi-agent proximal policy optimization (MAPPO) training framework based on an adaptive course scheduler that dynamically adjusts the environment difficulty parameter, action noise intensity, and policy entropy coefficient;

[0016] S6: Adopts a centralized training and distributed execution architecture to output collaborative control strategies through the Actor-Critic network;

[0017] The environmental difficulty parameter includes at least one of a maximum target speed, a number of obstacles, and a capture distance threshold.

[0018] Furthermore, the step S1 specifically includes:

[0019] First, a bounded confrontation simulation scenario was constructed, which included five friendly UAVs, one dynamic target UAV, and multiple islands and reefs as obstacles. The conditions for successful capture and mission termination were defined. The successful capture conditions must meet the following conditions simultaneously:

[0020] The distance between all the UAVs involved in the roundup and the target was smaller than the capture radius;

[0021] The deviation between the encirclement angle formed by adjacent unmanned boats and the ideal angle does not exceed the threshold;

[0022] The angle between the velocity vector of each unmanned boat and the line connecting the target does not exceed the set angle.

[0023] Furthermore, the kinematic models of the two unmanned boats are established:

[0024]

[0025] in, is the current position of the unmanned boat, v i 、ω i , ψ i They are respectively represented as speed, angular velocity, and heading, Δt is the simulation time step, and x max and y max are respectively the maximum length of the horizontal and vertical axes of the environment, the speed v i and angular velocity ω i Subject to constraints, i.e. av and a ω are acceleration and angular acceleration respectively.

[0026] Furthermore, the normalized observation space in step S2 specifically includes:

[0027] The observation space expression of our own unmanned boat is:

[0028]

[0029] Among them, the observation information vector of our unmanned boat to the target is

[0030]

[0031] in, Indicates the relative distance between the own UAV and the target in the x and y directions; is the angle of the target relative to the unmanned boat; is the relative speed of the target relative to the own UAV in the x and y directions; is the Euclidean distance between the UAV and the target.

[0032] Observation information vector of friendly unmanned boat For example

[0033]

[0034] in, Indicates the relative distance between two friendly UAVs in the x and y directions; is the European distance between two unmanned boats; is the angle between the velocity vector of UAV i and the line vector of UAV j.

[0035] The information vector of cooperative observation between unmanned boats is:

[0036]

[0037] in, is the mean of the remaining capture distance of the unmanned boat, It is the capture angle difference between the unmanned boat and the adjacent unmanned boat.

[0038] The observation information vector of the physical obstacle by the ranging sensor on the unmanned boat The observation information vector of the unmanned boat for itself is

[0039]

[0040] The values ​​of each element are represented as normalized horizontal and vertical coordinate values ​​and speed values, respectively.

[0041] At the same time, the normalized observation matrix of the target unmanned boat is expressed as

[0042] o T =[o u ,o obser ,o TW ,o self ]

[0043] Among them, the target's observation information vector of our unmanned boat is For example

[0044]

[0045] in, is the relative distance between the target UAV and the own UAV in the x and y directions; Φ Ti The relative position between the target unmanned boat and the own unmanned boat; is the relative speed of the UAV relative to the target in the x and y directions; is the European distance between the target and the own unmanned boat.

[0046] The information vector observed by the ranging sensor on board the boat is

[0047]

[0048] Its own observation information vector is

[0049]

[0050] The observation information vector of the target unmanned boat on its own islands and reefs is

[0051]

[0052] Among them, W x ,W y is the coordinate value of one's own islands and reefs; T x ,T y is the coordinate value of the target; W radius The radius of one's own islands and reefs.

[0053] Both the enemy and the enemy use the expected speed and expected angle to indirectly control the movement of the unmanned boat. Since the observation characteristics of the design are mainly the orthogonal components of the coordinates, the action space δ and the expected speed v are defined. exp and expected angle The expression is as follows:

[0054]

[0055] Among them, v max is the maximum speed of the unmanned boat; a x ,a yare two orthogonal independent control quantities in the x and y directions, δ=[a x ,a y ]; a x ,a y ∈[-1,1]. Finally, the unmanned boat motion control algorithm is introduced to indirectly control the acceleration a v and angular acceleration a ω , thereby simplifying the decision-making difficulty of the system.

[0056] Furthermore, the step S3 specifically includes:

[0057] In order to enable our own unmanned boats to quickly pursue the target and avoid obstacles at the same time, and finally complete the encirclement mission, our own unmanned boats share a set of reward functions. Through distance rewards, angle rewards, cooperation rewards, speed rewards, success rewards, collision penalties and time penalties, the unmanned boats gradually learn how to cooperate to capture the evader. The specific reward function is:

[0058] Distance Reward R dist Defined as

[0059]

[0060] Among them, d i is the distance between the unmanned boat i and the target; is the mean distance from all activated UAVs to the target; σ d is the standard deviation of the distance from all unmanned boats to the target (the minimum value is ε σ =0.01); k λ1 and k λ2 It is an adjustable reward coefficient.

[0061] Roundup Angle Bonus R angle Defined as:

[0062]

[0063] in, is the ideal angle; α ij and α ik are the angles formed by the unmanned boat i and its left and right adjacent unmanned boats relative to the target; Δα i =|α ij -α ik | is the angle difference; k a1 and k a2 It is an adjustable reward coefficient.

[0064] Speed ​​Bonus R vel Defined as

[0065]

[0066] Among them, v i is the current speed of the unmanned boat i; v max Its maximum speed; k v It is an adjustable reward coefficient.

[0067] Synergy Reward R coop Defined as

[0068]

[0069] Among them, d i is the distance between the unmanned boat i and the target; d cap is the capture distance threshold.

[0070] Collision penalty function R coll Definition: Actual collision penalty P actual Determined by collision type

[0071]

[0072] Laser Warning Penalty R laser =l min -1, where l min ∈[0,1] is the minimum laser reading after normalization. The final collision penalty R coll is the minimum value of all possible penalties, that is, R coll =min(P actual ,P laser ).

[0073] Then the unmanned boat will receive a fixed time penalty R at each time step time = -0.02, aiming to complete the roundup task as quickly as possible. Then, the reward for a successful roundup is expressed as

[0074]

[0075] Among them, R capture This is a one-time success reward value, which is only given when a roundup is successful.

[0076] Finally, the total reward R obtained by the unmanned boat at each time step is step for

[0077]

[0078] Among them, w i are weight coefficients, and their sum is always 1; R ξ Reward R dist ,R angle ,R vel ,R coll A collection of .

[0079] At the same time, the total reward of the target is composed of five parts: the reward for approaching the workstation, the reward for avoiding, the collision penalty, the time step penalty, and the reward for reaching the workstation. time =-0.01, when the target reaches its own island or reef, a one-time positive incentive of 20 is given.

[0080] The reward for being close to a workstation is expressed as

[0081] R approach =K approach (Γ last -Γ current )

[0082] Among them, K approach is the reward coefficient; Γ last is the distance between the target and the island at the previous moment; Γ current It is the distance between the target and one's own islands and reefs at the current moment.

[0083] The avoidance reward encourages the target to effectively avoid the pursuer, which is expressed as

[0084] R evade =K evade (d min,current -d min,last )

[0085] Among them, K evade is the reward coefficient; d min,current is the distance between the target and the nearest unmanned boat at the current moment; d min,last The distance between the target and the nearest unmanned boat at the previous moment.

[0086] The collision penalty consists of three sub-items, and the most severe penalty is taken, which is expressed as R collision =min{R bnd ,R entity ,R laser}, which are boundary collision penalty, solid obstacle collision penalty, and laser obstacle avoidance penalty, respectively. Their designs are the same as those of the self-collision penalty function.

[0087] Furthermore, the step S4 specifically includes:

[0088] The decision-making process of the friendly UAV can be defined using a partially observable Markov decision process (Dec-POMDP). In a Dec-POMDP, each agent only has access to local observation information (including perception and communication) and lacks global state information. At each time step, each agent makes an action decision based on its local observation information, and all agents perform a joint action to update the environment state.

[0089] Dec-POMDP is defined as Where N is the number of unmanned boats participating in the encirclement and capture mission; S is the state space of the environment. It is a joint observation space for multiple unmanned boats. is the local observation space of the i-th unmanned boat. It is worth noting that for the unmanned boats performing the pursuit mission, the hunting game process is carried out in the local observation domain of each agent, which means that their decision-making is based on incomplete local information state, which makes Therefore, the pursuit process of one's own unmanned boat can also be described as is the joint action space of the unmanned boat cluster, where Represents the action space of the i-th unmanned boat. is the state transition probability, is the shared reward, i.e., the reward r(s,a) received by all unmanned boats in this state and they take joint action a. γ∈[0,1] represents the discount factor used to adjust the impact of future rewards on the current state.

[0090] The strategy π(a|s) of the unmanned boat is the probability function from state s to action a. The ultimate goal of the Markov decision process (MDP) is to find the strategy that maximizes the cumulative discounted reward R of each agent. i,t The optimal strategy π * . As shown in the formula.

[0091]

[0092] where r i,t+k is the immediate reward at step k.

[0093] Furthermore, the step S5 specifically includes:

[0094] An adaptive course scheduler is used to dynamically adjust the difficulty of the environment to achieve efficient and adaptable training of the agent. After each round of training, the scheduler calculates the success rate of the sliding window at the current difficulty. If the success rate is higher than the set threshold and the number of training rounds meets the minimum requirement, the scene difficulty is increased; if the success rate is lower than the degradation threshold, the scene difficulty is reduced. If the performance of the unmanned vehicle does not improve significantly over a long period of time (i.e., stagnation), a degradation signal will also be triggered to prevent the strategy from falling into a local optimum. The core expression of its difficulty update is:

[0095]

[0096] Among them, d t is the current difficulty; is the sliding window success rate; is the upgrade threshold; The number of training rounds for the current difficulty level; is the minimum number of training rounds; λ is the degradation coefficient; is the stagnation counter; is the patience threshold, which is the maximum number of stalls allowed.

[0097] At the same time, in order to promote the exploration of one's own unmanned boat and prevent it from falling into the local optimum, this study adds noise to the action of each unmanned boat, which is expressed as

[0098] a i =π θ (o i )+σ t ·z

[0099] Among them, z is a standard normal distribution sample, and the noise scale σ t Decays linearly with the progress of course-based training.

[0100] In addition, the target's maneuvering strategy is mainly generated by the PPO algorithm. During the training process, the specific action value is generated based on the adaptive hybrid strategy, and its expression is

[0101] π mix (a|s)=β·π PPO (a|s)+(1-β)·π random (a|s)

[0102] Among them, π PPO (a|s) is the action strategy generated by the PPO algorithm; π random (a|s) is a randomly generated action strategy; β is dynamically adjusted with the difficulty level, and its expression is:

[0103]

[0104] Among them, d is the current difficulty level, d max The maximum difficulty level.

[0105] Furthermore, the step S6 specifically includes:

[0106] The MAPPO algorithm adopts a centralized training and distributed execution architecture. Each unmanned boat is trained based on local observations. i Action a is independently selected by the policy network i, achieving decentralized execution. The policy network input is the flattened one-dimensional real-valued vector of each UAV's normalized 44-dimensional local observation vector. Its hidden layer consists of two 256-node fully connected layers, each followed by a Reinforced Luminaire (ReLU) activation function. Its output is the mean and standard deviation of the continuous action space, which is then clamped to a range between -1 and 1 using a Tanh activation function. The UAV's specific actions are sampled from a normal distribution with the mean and standard deviation as parameters, and then normalized again using a Tanh activation function. However, the value network estimates the global value based on the global state S and adjusts the policy network parameters, thus achieving centralized training. The value network input is the flattened one-dimensional real-valued vector of the 132-dimensional global state vector concatenated from the local observation vectors of all activated UAVs. It also consists of two 256-node fully connected layers and a Reinforced Lu (Reinforced Lu) activation function. Its output layer is a linear transformation, directly outputting a scalar that estimates the global value of the current joint state, thereby facilitating collaborative decision-making among multiple agents. The policy network optimization goal is to maximize the objective function subject to the proximal policy optimization constraints:

[0107]

[0108] Among them, B is the batch size; n is the number of unmanned boats participating in the roundup; is the probability ratio of the new and old strategies; is the estimated advantage function; ∈ is the clipping parameter, which is used to limit the strategy update amplitude to avoid excessive strategy changes that lead to unstable training; is the policy entropy, α H is the entropy coefficient.

[0109] The value network optimization uses mean squared error loss and adds value clipping to enhance stability, with the goal of minimizing the loss function. The loss function expression is as follows.

[0110]

[0111] in, Represents the value network of the state estimated value of; Represents the calculated target return value.

[0112] The target strategy is mainly trained by the single-agent proximal policy optimization algorithm (PPO). Its policy network and value network are both composed of fully connected layers, and the network structure is the same as that of its own unmanned boat.

[0113] In order to effectively measure the status of the unmanned boat t Take action a tThis study uses an advantage function, namely the generalized advantage estimate (GAE), to evaluate the performance of the unmanned vehicle relative to the average performance. The main calculation formula is as follows:

[0114]

[0115] Among them, δ t =r t +γV(s t+1 )-V(s t ) is the temporal difference error; γ is the discount factor; λ is the GAE parameter used to balance the bias and variance.

[0116] The MAPPO experience pool stores the joint observation data of each step in the UAV round. The PPO experience pool stores the observation data of the target unmanned boat. t ,a t ,r t ,o t+1 During training, a policy update is performed every time a preset trajectory length is reached. Multiple rounds of policy and value network updates are performed on each batch of data. After the update, the experience pool is cleared, strictly adhering to the principles of online learning.

[0117] Beneficial effects of the present invention:

[0118] This paper addresses the issues of simplistic scenarios, low training efficiency, sparse rewards, and poor generalization in unmanned boat capture and confrontation tasks, and significantly improves performance through the following innovative designs:

[0119] (1) Environment construction and modeling innovation:

[0120] Construct a two-dimensional ocean confrontation simulation scenario that includes dynamic targets, multiple island and reef obstacles, and our islands and reefs;

[0121] Set multi-dimensional capture success conditions (distance / angle / speed coordinated constraints) to truly simulate the complexity of the confrontation task;

[0122] The multi-UAV encirclement confrontation is modeled as a Dec-POMDP problem to accurately describe the collaborative decision-making process under local observations.

[0123] (2) Adaptive course scheduling mechanism:

[0124] Design a dynamic difficulty adjustment strategy: automatically upgrade or downgrade the environment difficulty based on the sliding window success rate, number of training rounds, and stagnation counter;

[0125] Dynamically adjust the target's maximum speed, number of obstacles, and capture distance threshold;

[0126] Balancing exploration and exploitation through action noise attenuation and target mixing strategies.

[0127] (3) Multi-agent training framework optimization:

[0128] Centralized training and decentralized execution architecture using MAPPO algorithm;

[0129] Design multi-scale reward function (distance / angle / speed / collaborative reward + collision / success penalty);

[0130] Optimize policy gradient calculation through generalized advantage estimation (GAE).

[0131] In summary, the present application effectively solves the problems of low training efficiency, sparse reward and poor generalization in multi-unmanned vehicle cooperative hunting, and significantly improves the cooperative decision-making ability and adversarial game efficiency of the agent. BRIEF DESCRIPTION OF DRAWINGS

[0132] Figure 1 is the overall flowchart of the method described in the present application;

[0133] Figure 2 is the unmanned vehicle coordinate system schematic diagram provided by the present application embodiment 1;

[0134] Figure 3 is the unmanned vehicle sensor detection model schematic diagram provided by the present application embodiment 1;

[0135] Figure 4 is the own unmanned vehicle observation model schematic diagram provided by the present application embodiment 1;

[0136] Figure 5 is the target unmanned vehicle observation model schematic diagram provided by the present application embodiment 1;

[0137] Figure 6 is the highest difficulty scenario schematic diagram provided by the present application embodiment 2;

[0138] Figure 7 is the cumulative reward value curve comparison diagram of different algorithms provided by the present application embodiment 2. DETAILED DESCRIPTION

[0139] In order for those skilled in the art to fully understand and implement the present application, the preferred embodiments of the present application will be described in detail in conjunction with the drawings. It should be understood that the following examples are only used to illustrate the present application, but not to limit the present application.

[0140] Example 1

[0141] In conjunction with Figure 1 The present embodiment describes an unmanned vehicle hunting adversarial decision-making method based on adaptive curriculum reinforcement learning, which specifically includes the following steps:

[0142] S1: Construct a two-dimensional marine confrontation simulation scene, including dynamic targets, multiple island obstacles and our island, set the successful capture condition and task termination condition, and establish the kinematics model of unmanned ship;

[0143] S2: Design a normalized state space and a continuous action space based on expected speed and angular velocity;

[0144] S3: Design a multi-scale reward function, including distance reward, angle coordination reward, speed reward, collision penalty and task completion reward;

[0145] S4: Model the multi-unmanned ship capture confrontation as a Markov game problem;

[0146] S5: Construct a MAPPO training framework based on an adaptive curriculum scheduler, which dynamically adjusts the environmental difficulty parameters (target maximum speed, obstacle number, capture distance threshold), unmanned ship action noise intensity and action policy entropy coefficient;

[0147] S6: Use a centralized training and distributed execution architecture to output a cooperative control strategy through an Actor-Critic network.

[0148] Below, reference will be made to Figures 1 to 6 The above adaptive curriculum reinforcement learning-based unmanned ship capture confrontation decision-making method in the present example embodiment will be described in more detail. The step S1 specifically includes:

[0149] First, a bounded confrontation simulation scene is constructed, which includes 5 of our unmanned ships, 1 dynamic target unmanned ship and multiple islands as obstacles, the capture success condition and the task termination condition are defined, and the capture success condition needs to meet the following conditions at the same time:

[0150] At least one unmanned unit is less than the capture radius from the target;

[0151] The surrounding angle formed between adjacent unmanned units does not deviate from the ideal angle by more than a threshold value;

[0152] The angle between the velocity vector of each unmanned unit and the connecting line of the target does not exceed a set angle.

[0153] Further, the kinematics model of the unmanned ships of both parties is established:

[0154]

[0155] wherein, is the current position of the unmanned ship, v i , ω i , ψ i represent the speed, angular velocity and heading respectively, Δt is the simulation time step, x max and y maxThey are respectively the maximum length of the horizontal and vertical axes of the environment, the speed v i and angular velocity ω i Subject to constraints, i.e. a v and a ω are acceleration and angular acceleration respectively.

[0156] In this embodiment, step S2 specifically includes:

[0157] The observation space expression of our own unmanned boat is:

[0158]

[0159] Among them, the observation information vector of our unmanned boat to the target is

[0160]

[0161] in, Indicates the relative distance between the own UAV and the target in the x and y directions; is the angle of the target relative to the unmanned boat; is the relative speed of the target relative to the own UAV in the x and y directions; is the Euclidean distance between the UAV and the target. The observation information vector of the friendly UAV is For example

[0162]

[0163] in, Indicates the relative distance between two friendly UAVs in the x and y directions; is the European distance between two unmanned boats; is the angle between the velocity vector of UAV i and the line vector of UAV j.

[0164] The information vector of cooperative observation between unmanned boats is:

[0165]

[0166] in, is the mean of the remaining capture distance of the unmanned boat, It is the capture angle difference between the unmanned boat and the adjacent unmanned boat.

[0167] The observation information vector of the physical obstacle by the ranging sensor on the unmanned boat The observation information vector of the unmanned boat for itself is

[0168]

[0169] The values ​​of each element are represented as normalized horizontal and vertical coordinate values ​​and speed values, respectively.

[0170] At the same time, the normalized observation matrix of the target unmanned boat is expressed as

[0171] o T =[o u ,o obser ,o TW ,o self ]

[0172] Among them, the target's observation information vector of our unmanned boat is For example

[0173]

[0174] in, is the relative distance between the target UAV and the own UAV in the x and y directions; Φ Ti The relative position between the target unmanned boat and the own unmanned boat; is the relative speed of the UAV relative to the target in the x and y directions; is the European distance between the target and the own unmanned boat.

[0175] The information vector observed by the ranging sensor on board the boat is

[0176]

[0177] Its own observation information vector is

[0178]

[0179] The observation information vector of the target unmanned boat on its own islands and reefs is

[0180]

[0181] Among them, W x ,W y is the coordinate value of one's own islands and reefs; T x ,T y is the coordinate value of the target; W radius The radius of one's own islands and reefs.

[0182] Both the enemy and the enemy use the expected speed and expected angle to indirectly control the movement of the unmanned boat. Since the observation characteristics of the design are mainly the orthogonal components of the coordinates, the action space δ and the expected speed v are defined. exp and expected angle The expression is as follows:

[0183]

[0184] Among them, v max is the maximum speed of the unmanned boat; a x ,a y are two orthogonal independent control quantities in the x and y directions, δ=[a x ,a y ]; a x ,a y ∈[-1,1]. Finally, the unmanned boat motion control algorithm is introduced to indirectly control the acceleration a v and angular acceleration a ω , thereby simplifying the decision-making difficulty of the system.

[0185] In this embodiment, step S3 specifically includes:

[0186] In order to enable our own unmanned boats to quickly pursue the target and avoid obstacles at the same time, and finally complete the encirclement mission, our own unmanned boats share a set of reward functions. Through distance rewards, angle rewards, cooperation rewards, speed rewards, success rewards, collision penalties and time penalties, the unmanned boats gradually learn how to cooperate to capture the evader. The specific reward function is:

[0187] Distance Reward R dist Defined as

[0188]

[0189] Among them, d i is the distance between the unmanned boat i and the target; is the mean distance from all activated unmanned boats to the target; σ d is the standard deviation of the distance from all unmanned boats to the target (the minimum value is ε σ =0.01); k λ1 and k λ2 It is an adjustable reward coefficient.

[0190] Roundup Angle Bonus R angle Defined as:

[0191]

[0192] in, is the ideal angle; α ij and α ik are the angles formed by the unmanned boat i and its left and right adjacent unmanned boats relative to the target; Δα i =|α ij -α ik | is the angle difference; k a1 and k a2 It is an adjustable reward coefficient.

[0193] Speed ​​Bonus R vel Defined as

[0194]

[0195] where v i is the current speed of the USV i; v max is its maximum speed; k v is an adjustable reward coefficient.

[0196] The cooperative reward R coop is defined as

[0197]

[0198] where d i is the distance between the USV i and the target; d cap is the capture distance threshold. The collision penalty function R coll is defined as the actual collision penalty P actual is determined according to the collision type as

[0199]

[0200] The laser warning penalty R laser = l min - 1, where l min ∈ [0, 1] is the normalized minimum laser reading. The final collision penalty R coll is the minimum among all possible penalties, i.e., R coll = min(P actual , P laser ).

[0201] The USV then receives a fixed time-consuming penalty R time = -0.02 at each time step, aiming to complete the pursuit task as soon as possible. Then, the reward for a successful pursuit is represented as

[0202]

[0203] where R capture is a one-time success reward value, which is only given at the moment of successful pursuit.

[0204] Finally, the total reward R step received by the USV at each time step is

[0205]

[0206] where w i are weight coefficients, and their sum is always 1; R ξ represents the reward R dist , R angle , R vel , Rcoll a set of targets.

[0207] Meanwhile, the total reward of the target is composed of five parts: the approaching work station reward, the evading reward, the collision penalty, the time step penalty and the work station arrival reward. Among them, the time step penalty R time =-0.01, when the target arrives at the own island, a one-time positive incentive is given, and the value is 20.

[0208] The approaching work station reward is expressed as

[0209] R approach =K approach (Γ last -Γ current )

[0210] Wherein, K approach is the reward coefficient; Γ last is the distance between the target and the own island at the last time; Γ current is the distance between the target and the own island at the current time.

[0211] The evading reward encourages the target to effectively avoid the pursuer, and its expression is

[0212] R evade =K evade (d min,current -d min,last )

[0213] Wherein, K evade is the reward coefficient; d min,current is the distance between the target and the nearest unmanned ship at the current time; d min,last is the distance between the target and the nearest unmanned ship at the last time.

[0214] The collision penalty is composed of 3 sub-items, and the most serious penalty is taken, and its expression is R collision =min{R bnd ,R entity ,R laser}, which are boundary collision penalty, solid obstacle collision penalty and laser obstacle avoidance based penalty, which are designed in the same way as the own collision penalty function.

[0215] In the embodiment, the step S4 specifically comprises:

[0216] The decision-making process of the own unmanned ship can be defined using a partially observable Markov decision process (Dec-POMDP). In Dec-POMDP, each agent can only obtain local observation information (including perception and communication), and cannot obtain global state information. At each time step, each agent makes action decision according to its local observation information, and all agents execute joint action to refresh the environment state.

[0217] Dec-POMDP is defined as Where N is the number of unmanned boats participating in the encirclement and capture mission; S is the state space of the environment. It is a joint observation space for multiple unmanned boats. is the local observation space of the i-th unmanned boat. It is worth noting that for the unmanned boats performing the pursuit mission, the hunting game process is carried out in the local observation domain of each agent, which means that their decision-making is based on incomplete local information state, which makes Therefore, the pursuit process of one's own unmanned boat can also be described as is the joint action space of the unmanned boat cluster, where Represents the action space of the i-th unmanned boat. is the state transition probability, is the shared reward, i.e., the reward r(s,a) received by all unmanned boats in this state and they take joint action a. γ∈[0,1] represents the discount factor used to adjust the impact of future rewards on the current state.

[0218] The strategy π(a|s) of the unmanned boat is the probability function from state s to action a. The ultimate goal of the Markov decision process (MDP) is to find the strategy that maximizes the cumulative discounted reward R of each agent. i,t The optimal strategy π * . As shown in the formula.

[0219]

[0220] where r i,t+k is the immediate reward at step k.

[0221] In this embodiment, step S5 specifically includes:

[0222] An adaptive course scheduler is used to dynamically adjust the difficulty of the environment to achieve efficient and adaptable training of the agent. After each round of training, the scheduler calculates the success rate of the sliding window at the current difficulty. If the success rate is higher than the set threshold and the number of training rounds meets the minimum requirement, the scene difficulty is increased; if the success rate is lower than the degradation threshold, the scene difficulty is reduced. If the performance of the unmanned vehicle does not improve significantly over a long period of time (i.e., stagnation), a degradation signal will also be triggered to prevent the strategy from falling into a local optimum. The core expression of its difficulty update is:

[0223]

[0224] Among them, d t is the current difficulty; is the sliding window success rate; is the upgrade threshold; is the current difficulty training round number; is the minimum training round number; λ is the degradation coefficient; is the stagnation counter; is the patience threshold, that is, the maximum number of allowed stagnations.

[0225] At the same time, in order to promote the exploration of the own unmanned vehicle and prevent falling into local optimum, the research adds noise to the action of each unmanned vehicle, and the expression is

[0226] a i = π θ (o i )+ σ t · z

[0227] wherein z is a standard normal distribution sample, and the noise scale σ t linearly decays with the course training process.

[0228] In addition, the target maneuvering strategy is mainly generated by the PPO algorithm, and in the training process, the specific action value is generated based on the adaptive mixed strategy, and the expression is

[0229] π mix (a|s) = β· π PPO (a|s) + (1-β)· π random (a|s)

[0230] wherein π PPO (a|s) is the action strategy generated by the PPO algorithm; π random (a|s) is the randomly generated action strategy, and the expression is β dynamically adjusts with the difficulty level, and the expression is:

[0231]

[0232] wherein d is the current difficulty level, and d max is the maximum difficulty level.

[0233] In the embodiment, the step S6 specifically comprises:

[0234] The MAPPO algorithm adopts the architecture of centralized training and decentralized execution, and each unmanned vehicle independently selects the action a i through the policy network according to the local observation o i, the input of the policy network is the normalized 44-dimensional local observation vector of each USV flattened into a one-dimensional real value vector, and its hidden layer consists of two fully connected layers with 256 nodes in each layer, and Relu activation function is used after each layer. The output is the mean and standard deviation of the continuous action space, and the mean is limited between -1 and 1 through the Tanh activation function. The specific action of the USV is obtained by sampling the normal distribution with the mean and standard deviation as parameters, and then normalized by the Tanh activation function again. But the value network estimates the global value according to the global state S, and adjusts the parameters of the policy network, so as to realize centralized training. The input of the value network is the spliced local observation vector of all activated USVs into a 132-dimensional global state vector flattened into a one-dimensional real value vector, which also contains two fully connected layers with 256 nodes and Relu activation function. Its output layer is a linear transformation, which directly outputs a scalar to estimate the global value of the current joint state, so as to promote the collaborative decision-making among multiple agents. The optimization goal of the policy network is to maximize the objective function constrained by the proximal policy optimization:

[0235]

[0236] Where B is the batch size; n is the number of USVs participating in the pursuit; is the probability ratio of the new and old policies; is the estimated advantage function; ∈ is the clipping parameter, which is used to limit the policy update amplitude and avoid unstable training caused by too large policy change; is the policy entropy, and α H is the entropy coefficient.

[0237] The value network optimization adopts mean square error loss, and adds value clipping to enhance stability, the purpose is to minimize the loss function. Its loss function expression is as follows.

[0238]

[0239] Where, represents the estimated value of the value network to the state ; represents the calculated target return value.

[0240] The target policy is mainly trained by the single-agent proximal policy optimization algorithm (PPO), and its policy network and value network are composed of fully connected layers, and the network structure is the same as that of the USV.

[0241] In order to effectively measure the performance of the USV in the state o t , the action a tTo evaluate the actions of the USV, the study adopts a merit function, i.e., generalized advantage estimation (GAE) calculation, to evaluate the actions of the USV relative to the merits and demerits of the average performance. The main calculation formula is as follows:

[0242]

[0243] wherein δ t = r t + γV(s t+1 ) - V(s t ) is the time difference error; γ is the discount factor; and λ is the GAE parameter for balancing the bias and variance.

[0244] The MAPPO experience pool stores the joint observation data of the USV at each step in the round The PPO experience pool stores the observation data {o t ,a t ,r t ,o t+1} of the target USV. In the training, a policy update is performed every time the preset track length is reached, and multiple rounds of policy and value network updates are performed for each batch of data. After the update, the experience pool is emptied, and the principle of online learning is strictly followed.

[0245] Embodiment 2

[0246] The Python standard OpenCV library CV2 is used to build a two-dimensional battlefield environment, and the space class of the OpenAI Gym library is called to initialize the state space and action space of the USV. The construction and training of the neural network use the Pytorch framework, version 1.4.1. The hardware configuration is NVIDIA GeForce RTX GPU 4070Ti SUPER, processor Intel(R) Core(TM) i5-14600K, and memory 64 GB.

[0247] To ensure the rigor and reproducibility of the multi-USV cooperative hunting confrontation simulation experiment in the study, the experimental environment and the key parameters of each USV are set as follows: the simulation environment is built as a two-dimensional continuous rectangular area, the maximum range of the X-axis is set to 550 m, the maximum range of the Y-axis is set to 450 m, the time step of the entire simulation process is 0.5 s, and the maximum number of steps per round is 300. The scene includes 5 USVs and 1 target USV. The initial speed of each USV is 1 m / s, the maximum speed is 12 m / s, the maximum turning angular velocity is about 0.611 rad / s, and the maximum linear acceleration is 4 m / s 2The safety distance between the UAVs is set to 10m, and a fixed initial deployment position and orientation is used. At the beginning of the mission, the system will activate the three UAVs closest to the target UAV to participate in the pursuit. The initial speed of the target UAV is set to 1.5m / s, its maximum turning angular velocity is 0.524rad / s, and its maximum linear acceleration is 4m / s. 2 , whose initial position is randomly generated around the cordon.

[0248] The most difficult scenario in this experiment is Figure 6 shown.

[0249] The environmental parameters that are dynamically adjusted with the difficulty in the adaptive course learning mechanism include the maximum speed of the target, the number of obstacles, the capture distance of our UAV, and the initial position of the target. Note that obstacles are represented by b1, b2, and b3, and their detailed configurations are shown in Tables 1 and 2.

[0250] Table 1 Parameter configuration of adaptive course learning scheduler

[0251]

[0252] Table 2 Parameter configuration of adaptive course-based learning environment

[0253]

[0254]

[0255] In particular, the following explanations are given for the experimental setup: all friendly UAVs are isomorphic, that is, the performance parameters of all UAVs are exactly the same, and a fully connected communication network is used between friendly UAVs; the simulation environment of the experiment is a bounded area, and to ensure the existence of a feasible solution, only the encirclement and confrontation situation in which the maximum speed of the friendly UAV is higher than the maximum speed of the target UAV is considered.

[0256] In order to demonstrate the advantages of the method of the present invention, the no-course MAPPO (NOCL-MAPPO) algorithm, the traditional course MAPPO (CL-MAPPO) algorithm and the method of the present invention (ATCL-MAPPO) are compared. Figure 7 As shown in the figure, the ATCL-MAPPO algorithm quickly achieves a high reward value early in training, which continues to grow steadily, ultimately converging to a reward value of approximately 150, significantly outperforming the other two algorithms. While the CL-MAPPO algorithm's reward curve also shows an upward trend, its growth rate is slower, and its final converged reward value is lower than that of ATCL-MAPPO, at approximately 100 to 125. The NOCL-MAPPO algorithm's reward curve remains low throughout training, with significant fluctuations, indicating poor learning performance.

[0257] The preferred embodiments of the application disclosed above are only to facilitate the elucidation of the application. The preferred embodiments do not describe all the details of the application and limit the application to the specific embodiments described. Obviously, many modifications and variations can be made in light of the teachings above. The description is chosen and described in order to best explain the principles of the application and its practical application to thereby enable others skilled in the art to best utilize the application and get the best results from the application. The application is only limited by the claims and their full scope and equivalents.

Claims

1. The unmanned boat encirclement and capture confrontation decision-making method based on adaptive curriculum reinforcement learning is characterized by: The following steps are involved: S1: Construct a two-dimensional ocean confrontation simulation scenario including dynamic targets, obstacles, and friendly islands and reefs, set the conditions for successful capture and mission termination, and establish a kinematic model for the unmanned boat; S2: Design the normalized state space and the continuous action space based on the expected speed and angular velocity; S3: Design a multi-scale reward function that includes distance reward, angle coordination reward, speed reward, collision penalty and task completion reward; S4: Model the multi-UAV capture confrontation as a Markov game problem; S5: Build a multi-agent proximal policy optimization (MAPPO) training framework based on an adaptive course scheduler that dynamically adjusts the environment difficulty parameter, action noise intensity, and policy entropy coefficient; S6: Adopts a centralized training and distributed execution architecture to output collaborative control strategies through the Actor-Critic network; The environmental difficulty parameter includes at least one of a maximum target speed, a number of obstacles, and a capture distance threshold.

2. The unmanned boat capture confrontation decision-making method based on adaptive curriculum reinforcement learning according to claim 1 is characterized in that: The conditions for successful capture in step S1 must all meet the following conditions: The distance between all the UAVs involved in the roundup and the target is less than the capture radius; The deviation between the encirclement angle formed by adjacent unmanned boats and the ideal angle does not exceed a predetermined threshold; The angle between the velocity vector of each unmanned boat and the line connecting the target does not exceed the set angle; The kinematic model of the unmanned boat satisfies: in, is the current position of the unmanned boat, v i 、ω i , ψ i They are respectively represented as speed, angular velocity, and heading, Δt is the simulation time step, and x max and y max are respectively the maximum length of the horizontal and vertical axes of the environment, the speed v i and angular velocity ω i Subject to constraints, i.e. a v and a ω are acceleration and angular acceleration respectively.

3. The unmanned boat capture confrontation decision-making method based on adaptive curriculum reinforcement learning according to claim 1 is characterized in that: The normalized state space in step S2 includes: The observation space expression of our own unmanned boat is: Among them, the observation information vector of our unmanned boat to the target is in, Indicates the relative distance between the own UAV and the target in the x and y directions; is the angle of the target relative to the unmanned boat; is the relative speed of the target relative to the own UAV in the x and y directions; is the Euclidean distance between the UAV and the target; Define action space δ and expected speed v exp and expected angle The expression is as follows: Among them, v max is the maximum speed of the unmanned boat; a x ,a y are two orthogonal independent control quantities in the x and y directions, δ=[a x ,a y ]; a x ,a y ∈[-1,1]; Finally, the unmanned boat motion control algorithm is introduced to indirectly control the acceleration a v and angular acceleration a ω , thereby simplifying the decision-making difficulty of the system.

4. The unmanned boat capture confrontation decision-making method based on adaptive curriculum reinforcement learning according to claim 1 is characterized in that: The multi-scale reward function in step S3 includes: Distance Reward R dist Defined as Among them, d i is the distance between the unmanned boat i and the target; is the mean distance from all activated UAVs to the target; σ d is the standard deviation of the distance from all unmanned boats to the target (the minimum value is ε σ =0.01); k λ1 and k λ2 is an adjustable reward coefficient; Roundup Angle Bonus R angle Defined as: in, is the ideal angle; α ij and α ik are the angles formed by the unmanned boat i and its left and right adjacent unmanned boats relative to the target; Δα i =|α ij -α ik | is the angle difference; k a1 and k a2 is an adjustable reward coefficient; Speed ​​Bonus R vel Defined as Among them, v i is the current speed of the unmanned boat i; v max Its maximum speed; k v is an adjustable reward coefficient; Synergy Reward R coop Defined as Among them, d i is the distance between the unmanned boat i and the target; d cap is the capture distance threshold; Collision penalty function R coll Definition: Actual collision penalty P actual Determined by collision type Laser Warning Penalty P laser =l min -1, where l min ∈[0,1] is the minimum laser reading after normalization. The final collision penalty R coll is the minimum value of all possible penalties, that is, R coll =min(P actual ,P laser ); Finally, the total reward expression obtained by the unmanned boat at each time step is: The sum of the weight coefficients is 1.

5. The unmanned boat capture confrontation decision-making method based on adaptive curriculum reinforcement learning according to claim 1 is characterized in that: The step S4 is specifically as follows: Modeled as Dec-POMDP, Dec-POMDP is defined as Where N is the number of unmanned boats participating in the encirclement and capture mission; S is the state space of the environment. It is a joint observation space for multiple unmanned boats. is the local observation space of the i-th unmanned boat; The strategy π(a|s) of the unmanned boat is the probability function from state s to action a; the ultimate goal of the Markov decision process (MDP) is to find the maximum cumulative discounted reward R for each agent. i,t The optimal strategy π * ; As shown in the formula. where r i,t+k is the immediate reward at step k.

6. The unmanned boat capture confrontation decision-making method based on adaptive curriculum reinforcement learning according to claim 1 is characterized in that: The adaptive course scheduler in step S5 adjusts the difficulty level through the following mechanisms: When the sliding window success rate is higher than the upgrade threshold and the number of training rounds reaches the minimum, the difficulty is increased; When the success rate falls below the degradation threshold or the performance stagnation count exceeds the patience threshold, the difficulty is reduced; The difficulty update function satisfies: Among them, d t is the current difficulty; is the sliding window success rate; is the upgrade threshold; The number of training rounds for the current difficulty level; is the minimum number of training rounds; λ is the degradation coefficient; is the stagnation counter; is the patience threshold, i.e. the maximum number of stalls allowed; Action noise press a i =π θ (o i )+σ t z linearly decays; where z is a standard normal distribution sample and the noise scale σ t Linear decay as the course training progresses; At the same time, the target's maneuvering strategy is mainly generated by the PPO algorithm. During the training process, the specific action value is generated based on the adaptive hybrid strategy, and its expression is p mix (a|s)=β·π PPO (a|s)+(1-β)·π random (a|s) Among them, π PPO (a|s) is the action strategy generated by the PPO algorithm; π random (a|s) is a randomly generated action strategy; β is dynamically adjusted with the difficulty level, and its expression is: Among them, d is the current difficulty level, d max The maximum difficulty level.

7. The unmanned boat capture confrontation decision-making method based on adaptive curriculum reinforcement learning according to claim 1 is characterized in that: The MAPPO framework implementation in step S6 includes: Policy network: Input 44-dimensional local observation vector and output action distribution parameters normalized by Tanh; Value network: takes as input a 132-dimensional global state vector and outputs a scalar value estimate; The strategy optimization objective function is: Among them, B is the batch size; n is the number of unmanned boats participating in the roundup; is the probability ratio of the new and old strategies; is the estimated advantage function; ∈ is the clipping parameter, which is used to limit the strategy update amplitude to avoid excessive strategy changes that lead to unstable training; is the policy entropy, α H is the entropy coefficient; The value network loss function is: in, Represents the value network of the state estimated value of; Indicates the calculated target return value; The target's strategy is primarily trained using a single-agent Proximal Policy Optimization (PPO) algorithm. Its policy network and value network are both composed of fully connected layers, and the network structure is the same as that of the enemy's unmanned boat. In order to effectively measure the status of the unmanned boat t Take action a t The superiority function, namely the generalized advantage estimate (GAE), is used to evaluate the performance of the unmanned vehicle relative to the average performance. The main calculation formula is as follows: Among them, δ t =r t +γV(s t+1 )-V(s t ) is the temporal difference error; γ is the discount factor; λ is the GAE parameter used to balance the bias and variance.

Images