A cooperative hunting method based on positive vortex potential field and differentiable quadratic programming

Abstract

The application discloses a kind of based on positive vortex potential field and the cooperative hunting method of differentiable quadratic programming, comprising: obtaining each pursuit intelligent agent local observation state and input policy network, obtain original action intention;High-order control barrier function constraint is constructed;According to local potential field information, it is judged whether there is deadlock risk, if yes, then the tangential vortex component of positive vortex artificial potential field and original action intention are fused as reference action, if not, then original action intention is used as reference action;Reference action is input into differentiable quadratic programming solver, and safe action is solved under constraint and is executed;According to the state after executing safe action, it is calculated that shaping reward is superimposed with environment reward;According to state and total reward, construct value gradient, it is propagated to policy network and parameter is updated by solver in reverse, and iteration is until hunting is successful.The application gives consideration to the safety of multi-agent system and the consistency of policy learning.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent cooperative control technology, and particularly relates to a cooperative trapping method based on positive eddy current potential field and differentiable quadratic programming. Background Technology

[0002] Multi-agent cooperative encirclement refers to a typical multi-agent cooperative control task in which multiple pursuing agents cooperate to confine dynamic or static targets within a specific geometric area, preventing their escape. This type of task has significant application value in fields such as military confrontation, autonomous search and rescue, and traffic control. In multi-agent cooperative encirclement tasks in environments with dense obstacles, existing technologies mainly face the following technical challenges.

[0003] I. The problem of low exploration efficiency caused by sparse rewards; Cooperative encirclement tasks exhibit significant sparse reward characteristics, where agents only receive positive rewards when achieving a specific geometric encirclement configuration or capturing the target. If reinforcement learning is used alone for unguided exploration in a high-dimensional state space, numerous ineffective trials will lead to learning stagnation and slow convergence in the early stages of training, making it difficult to form a stable policy.

[0004] To alleviate the exploration dilemma, existing technologies mostly employ reward shaping and prior knowledge guidance. Among them, potential function-based reward shaping (PBRS) can provide guidance while keeping the optimal policy unchanged, but its effectiveness is highly dependent on the construction of the potential function. Improper design can easily lead to rewards deviating from the target or even triggering "reward hijacking". Although the artificial potential field method can intuitively provide attractive and repulsive guidance, the potential field coupling is complex in multi-agent collaborative scenarios, making it difficult to stably output a continuous guidance signal.

[0005] II. Local Minimum Problems in the Artificial Potential Field Method; Artificial potential field (APF) constructs a potential function by superimposing the gravitational field of the target and the repulsive field of the obstacle, guiding the agent to move along the gradient to achieve real-time obstacle avoidance and path planning. Recent research has combined it with deep reinforcement learning, using the heuristic guidance of attraction and repulsion to alleviate sparse rewards and improve obstacle avoidance performance.

[0006] However, the classic APF (Advanced Persistent Field Function) is prone to local minima in dense obstacle environments due to the superposition of potential fields, causing agents to stagnate or oscillate. In multi-agent cooperative trapping scenarios, the potential field coupling between agents and between agents and dynamic targets is more complex, and the mutual cancellation of attractive and repulsive forces weakens the guidance effect and cooperative stability. Existing improvement strategies (such as setting virtual target points, introducing random perturbations, or attenuating repulsive forces) are mostly designed for single agents and are difficult to effectively solve the local minima problem under complex interactions of multiple agents.

[0007] III. The lack of security constraints on reinforcement learning strategies; Deep reinforcement learning can achieve autonomous obstacle avoidance and navigation without pre-set maps, but traditional methods only aim to maximize cumulative rewards and lack explicit safety constraints, which can easily lead to device damage and safety risks in unknown and complex scenarios. Garcia et al.'s review also points out that relying on blind trial and error for exploration poses a great risk in physically security-sensitive scenarios.

[0008] To construct explicit safety constraints, existing research typically models tasks as constrained Markov decision processes (CMDPs). Altman's CMDP theory, by introducing a cost function and a safety threshold, transforms safety control into a constrained policy optimization problem. Early studies often employed "soft constraint" mechanisms with negative rewards and penalties, but Gu et al. pointed out that when high capture rewards mask collision penalties, agents are prone to "speculative behavior" that ignores physical risks; furthermore, the solution algorithm based on the Lagrange multiplier method faces the problem of difficult-to-control update step size in continuous action spaces, still leading to a large number of violations in the early stages of training.

[0009] In recent years, the control barrier function (CBF) safety filtering mechanism, combined with control theory, has become an important research direction. Cheng et al. first introduced CBF into reinforcement learning, using a safety set that satisfies forward invariance and quadratic programming (QP) to perform online minimum correction of the policy output, balancing safety and the performance of the original policy. Subsequently, Emam et al. extended it to multi-agent systems (MACBF), achieving hard safety assurance in complex cooperative scenarios.

[0010] However, the aforementioned security filtering mechanism based on CBF-QP still has the following technical shortcomings: (i) The discontinuity of gradient connections between the security layer and the policy network is another core bottleneck of existing methods. The traditional quadratic programming (QP) solution process is usually non-differentiable, making it difficult for the policy network to obtain gradient information related to security adjustments, thus weakening its ability to perceive underlying physical constraints (Ma et al.). When the security layer significantly adjusts its actions, the policy update direction will deviate from the actual execution behavior; Wang et al. further pointed out that such control interventions disrupt behavioral consistency in complex cooperative scenarios, hindering the formation of stable cooperative policies. Especially in sparse reward environments, frequent interventions by the security layer further weaken effective exploration signals, significantly reducing the learning efficiency of cooperative trapping strategies.

[0011] (II) The Challenge of Representing Safety Constraints in High-Order Dynamical Systems. Multi-agent encirclement often employs a second-order integrator dynamics model (with acceleration as the control input). Since the position-based safety function has a second-order relative degree with respect to the control input, the standard first-order control obstacle function (CBF) is difficult to apply directly. Therefore, a higher-order control obstacle function (HOCBF) must be introduced to handle higher-order derivative relationships. How to uniformly represent the two types of hard safety constraints—collision avoidance and obstacle avoidance—within the HOCBF framework and transform them into optimization problems solvable online at the execution layer remains a key technical bottleneck that urgently needs to be addressed.

[0012] (III) Constraint Feasibility and Optimization Robustness in Dense Obstacle Scenarios. In environments with multiple dense obstacles, the safety intersection formed by multiple HOCBF constraints may be empty, causing the standard QP solver to fail. How to ensure that the optimization problem has a feasible solution in any state and maintain the numerical stability of the solution process through a reasonable relaxation mechanism while ensuring the strictness of safety constraints is a challenge that must be overcome in the engineering deployment of the safety layer.

[0013] In summary, existing technologies have at least the following shortcomings in multi-agent cooperative trapping tasks: (1) sparse rewards lead to low exploration efficiency, and existing reward shaping methods are highly sensitive to potential function construction, making it difficult to balance optimality preservation and guidance effects; (2) artificial potential field methods are prone to getting trapped in local minima in dense obstacle scenarios, and existing improved schemes are difficult to apply to complex interactive scenarios of multi-agent cooperative trapping; (3) the CBF-QP-based safety filter has gradient disconnection problems, causing high-level policies to be unable to perceive low-level safety corrections, and there is still a lack of effective solutions for high-order dynamic constraint representation and optimization robustness in dense scenarios. Therefore, it is urgent to provide a multi-agent cooperative trapping method that can simultaneously possess global dense guidance, local deadlock breaking, and safe differentiable execution capabilities to address the shortcomings of the aforementioned existing technologies. Summary of the Invention

[0014] To address the aforementioned technical problems, this invention proposes a collaborative trapping method based on positive eddy current potential field and differentiable quadratic programming, thereby resolving the issues present in the prior art.

[0015] To achieve the above objectives, this invention provides a cooperative trapping method based on a positive eddy current potential field and differentiable quadratic programming, comprising: Acquire local observation state information of each pursuit agent; The local observation state information is input into a pre-constructed policy network, and the policy network outputs the original action intentions of each pursuing agent. Based on the current state information and obstacle information of each pursuing agent, construct a high-order control obstacle function constraint; Based on the local potential field information of each pursuing agent, determine whether there is a risk of local deadlock. If so, fuse the tangential eddy current component of the positive eddy current artificial potential field with the original action intention to obtain a fused reference action; otherwise, use the original action intention as the reference action. The reference actions of each pursuing agent are input into a differentiable quadratic programming solver, and the safe actions of each pursuing agent are obtained and executed under the constraints of the higher-order control obstacle function. The state information of each pursuit agent after performing the safety action is obtained, and the shaping reward is calculated according to the reward shaping function based on the potential function. The environmental reward is obtained according to the environmental feedback. The shaping reward and the environmental reward are superimposed to obtain the total reward. Based on the state information of each pursuing agent after performing the safety action and the total reward, a value evaluation gradient is constructed. The value evaluation gradient is then backpropagated to the policy network through the differentiable quadratic programming solver to update the parameters of the policy network. The above steps are repeated until each pursuing agent meets the conditions for successful encirclement and capture, thus completing the cooperative encirclement and capture.

[0016] Optionally, the process of constructing higher-order control barrier function constraints includes: An obstacle avoidance constraint function is defined based on the positional relationship between the pursuing agent and the obstacle, and an inter-agent collision avoidance constraint function is defined based on the positional relationship between the pursuing agent and adjacent pursuing agents. The first derivative of the obstacle avoidance constraint function is used to obtain the corresponding first-order auxiliary function, and the first derivative of the inter-agent collision avoidance constraint function is used to obtain the corresponding first-order auxiliary function. Differentiating the first-order auxiliary function corresponding to the obstacle avoidance constraint function yields a second-order expression for the control input of the pursuit agent; differentiating the first-order auxiliary function corresponding to the inter-agent collision avoidance constraint function yields a second-order expression for the control input of the pursuit agent. Based on the second-order expression, the constraint coefficient matrix and constraint boundary vector are extracted, and the obstacle avoidance constraint and the inter-agent collision avoidance constraint are transformed into linear inequality constraints with respect to the control input.

[0017] Optionally, determining whether there is a risk of local deadlock based on the local potential field information of each of the pursuing agents includes: Calculate the resultant force vector of the target's gravitational direction vector and the obstacle's repulsive direction vector at the location of the pursuing agent; calculate the norm of the resultant force vector; when the pursuing agent enters the influence range of the obstacle and the norm of the resultant force vector is less than a preset gradient threshold, it is determined that there is a risk of local deadlock.

[0018] Optionally, the process of obtaining the tangential eddy current component of the positive eddy current artificial potential field includes: Obtain the target gravitational direction vector and the obstacle outward normal vector at the location of the pursuing agent; calculate the cross product between the target gravitational direction vector and the obstacle outward normal vector, and determine the rotation direction of the vortex field according to the sign of the cross product; construct an orthogonal rotation matrix according to the rotation direction; map the obstacle outward normal vector through the orthogonal rotation matrix to obtain the tangential vortex component.

[0019] Optionally, the tangential eddy current component of the positive eddy current artificial potential field is fused with the original action intention to obtain a fused reference action, including: The distance smoothing weight is calculated based on the distance between the pursuing agent and the obstacle, and the gradient smoothing weight is calculated based on the local potential field gradient norm. Multiplying the distance smoothing weight by the gradient smoothing weight yields the state-dependent deadlock activation coefficients. Determine whether the angle between the original intention of the action and the tangential eddy component is an obtuse angle. If so, peel off the conflicting component of the original intention of the action in the direction of the tangential eddy component to obtain the projected action. Based on the deadlock activation coefficient, the projection action and the tangential eddy current component are convexly combined to obtain the fused reference action.

[0020] Optionally, the reference actions of each pursuing agent are input into a differentiable quadratic programming solver to obtain the safe actions of each pursuing agent under the constraints of the higher-order control obstacle function, including: The reference actions of each of the aforementioned pursuit agents are combined with relaxation variables to form decision variables, wherein the relaxation variables are non-negative variables; The objective function is constructed by the quadratic term of the deviation between the reference action and the control action to be solved, and the sum of the linear penalty term and the quadratic regularization term of the relaxation variable; The decision variables are solved under the constraints of the higher-order control barrier function to obtain the safe actions of each pursuit agent and the corresponding values ​​of the relaxation variables.

[0021] Optionally, the process of constructing a reward shaping function based on a potential function includes: Construct a target gravitational potential energy function, which reaches its maximum value when the distance between the pursuing agent and the target is equal to the desired encirclement radius; Construct an obstacle repulsive potential energy function, wherein the obstacle repulsive potential energy function decreases as the distance decreases when the pursuing agent enters the influence range of the obstacle; Construct a trap configuration potential energy function, which reaches its maximum value when the angle difference between the pursuing agent and its neighboring pursuing agents relative to the target is equal to the desired angle interval; The composite potential energy function is obtained by weighted summing of the target gravitational potential energy function, the obstacle repulsive potential energy function, and the encirclement configuration potential energy function. The shaping reward is generated based on the difference between adjacent time steps of the composite potential function.

[0022] Optionally, it may also include: adding a takeover penalty to the total reward, the takeover penalty being calculated based on the deviation between the reference action and the safe action.

[0023] Optionally, backpropagating the value assessment gradient to the policy network via the differentiable quadratic programming solver includes: Construct a set of implicit function equations with Karush-Kuhn-Tucker conditions at the optimal solution of the differentiable quadratic programming solver; Based on the implicit function equations, the Jacobian matrix of the decision variable relative to the reference action is solved using the implicit function theorem. Based on the Jacobian matrix, the value evaluation gradient is backpropagated from the security action to the reference action, and then backpropagated from the reference action to the policy network.

[0024] Optionally, the parameters of the policy network are updated, including: Based on the status information of each of the pursuit agents after performing the safety action and the time difference error in the total reward calculation; Update the value network parameters based on the time difference error; The value assessment gradient is backpropagated to the policy network via the differentiable quadratic programming solver, and the policy network parameters are updated based on the backpropagated gradient. The target value network parameters and target policy network parameters are updated using a soft update method.

[0025] Compared with the prior art, the present invention has the following advantages and technical effects: This invention constructs a composite potential function comprising the target's gravitational potential energy, obstacle's repulsive potential energy, and encirclement configuration potential energy, and employs a reward shaping mechanism based on potential function difference. This provides a globally dense guiding signal while maintaining the original optimal policy, thus alleviating the exploration difficulties caused by sparse rewards and improving sample utilization efficiency in the early training stages. By introducing a positive eddy current artificial potential field, the tangential eddy current component provides obstacle-avoidance escape momentum in local minima, thereby breaking the deadlock state caused by the collinear balance of gravity and repulsion in traditional artificial potential field methods. This allows the agent to maintain a tendency to approach the target in dense obstacle environments. Furthermore, by embedding high-order control obstacle function constraints into a differentiable quadratic programming solver to construct a differentiable safety layer supporting gradient backpropagation, the high-level policy network can perceive the underlying physical safety boundary and obtain gradient information related to safety correction. This eliminates the deviation between the policy update direction and the actual execution behavior, avoids adversarial interaction between the policy network and the safety mechanism, and allows the policy to gradually and proactively adjust its action distribution during training, reducing dependence on posterior safety corrections. This invention combines a potential field guidance mechanism with a differentiable safety correction mechanism to ensure strict physical security of multi-agent systems while maintaining consistency between policy learning and underlying control. Attached Figure Description

[0026] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a diagram of the PI-STEP-MADDPG algorithm framework according to an embodiment of the present invention; Figure 2 This is a comparison chart of the total reward convergence curves for dynamic target capture in embodiments of the present invention; Figure 3 This is a schematic diagram illustrating the cumulative number of collisions for each algorithm in 20,000 training rounds according to an embodiment of the present invention; Figure 4 This is a comparison chart of the cumulative collision counts of various algorithms in 1000 test rounds according to embodiments of the present invention; Figure 5 This is a comparison chart of the dynamic target security intervention effects in an embodiment of the present invention. Detailed Implementation

[0027] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0029] Example 1 This embodiment provides a cooperative trapping method based on positive eddy current potential field and differentiable quadratic programming, including: (a) Modeling; (1) Foundations of security reinforcement learning modeling; (a) Continuous-time controlled affine dynamics model; Based on the discrete-time MARL dynamics model, this invention further introduces a continuous-time control affine model, laying the foundation for the construction of the safety space and related constraints for subsequent capture tasks. For the... A group of intelligent agents ( Its state equation can be expressed as: (1) in, Represents intelligent agents exist The state vector at any given moment. In a two-dimensional planar environment, the state of a second-order integrator model typically consists of two parts: position and velocity. Therefore, the agent's state vector... Can be defined as Vectors in space: (2) in, For position coordinates, For the velocity component. This represents the control input vector. In this study, the control input directly corresponds to the acceleration command or thrust command, i.e. . and It is a local Lipschitz continuous function that describes the inherent dynamic characteristics of the system.

[0030] (3) Internal drift item With control gain matrix Explicit expansion is as follows: (4) (b) The theory of control barrier functions; Based on the aforementioned dynamic model, to characterize the evolution of the safety function along the system trajectory and further construct safety constraints based on the control barrier function, Lie derivatives and related theories of the control barrier function are introduced. For any smooth scalar function... Its along the vector field and The Lie derivatives are defined as follows: (5) in, Describes the safety function of the system under no control input. The natural rate of change, and This characterizes the effect of the control input on the rate of change. For a general controlled affine system, if the constraint function... The relative order is Then its first The first derivative can be expressed as: (6) in, Indicates to Repetitive action The Lie derivative operator of the order of 1. Based on the Lie derivative, a control barrier function can be further defined to characterize the forward invariance of the safe set: if there exists a class function This makes it possible for all states All have control inputs satisfy: (7) but This inequality restricts the safety function, which is a control barrier function of the system. The decay rate is determined to ensure the forward invariance of the defined safe set when the corresponding conditions are met.

[0031] (c) Establishment of a safe space; This invention models the communication and sensing topology of the hunter swarm as an undirected graph. .in, express A set of nodes for each pursuer. The target to be captured is defined as Target, denoted as . intelligent agent The observation space adopts a local observation design, and its observation vector It mainly consists of its own state, relative information about the target, information about neighbor cooperation, and information about environmental obstacles.

[0032] The intelligent agent is first able to perceive its own physical state, including its current position. ,speed And its own serial number; within the detection radius of the airborne sensor. Inside, you can also obtain the target. Relative position is used relative to its own state information. and relative velocity To represent this, the intelligent agent also needs to utilize neighbor information to complete the modeling of cooperative behavior, defining the set of neighbor nodes. and using relative positions and relative velocity Describe the neighbors The state of the formation provides input for collision avoidance and geometric configuration maintenance. For environmental obstacle information, obstacle information is acquired using onboard sensors, and the LiDAR data is represented as a fixed-length ray range vector. ,in Indicates the first The distance to the nearest obstacle within each sector. In summary, the intelligent agent... observation vector It can be formalized as: (8) This invention employs a ring-shaped encirclement as the standard encirclement configuration. Effective encirclement is defined as follows: Effective encirclement of the target is achieved if and only if the following two geometric conditions are met. The first is the distance convergence condition: all agents participating in the encirclement converge to the target position. With the center as the target, the desired encirclement radius is... Within the circumferential neighborhood of radius , its mathematical expression is: (9) in, This represents the allowable distance error tolerance.

[0033] Then there's the relative velocity convergence condition; if only position convergence is satisfied... The agent may pass by the target at a high relative speed, causing the encirclement structure to become unstable. Therefore, further velocity cooperative constraints are added: (10) in, Let be the unit normal vector pointing from the agent to the target. This condition ensures that the agent's velocity in the radial direction is synchronized with the target. This is the allowable speed error tolerance.

[0034] Finally, there's the angular distribution condition. To maximize the blocking of the target's escape path, the agents used for the capture should be distributed as evenly as possible around the circumference. Let... For intelligent agents Relative to the target position, in a global context If the polar angle is measured with the positive axis as the reference direction, then the angular interval between adjacent agents is... Should meet: (11) in, The number of agents participating in the capture of this target. This is the tolerance for angle error.

[0035] The system must always satisfy two types of safety constraints: inter-agent collision avoidance (Equation 12) and obstacle avoidance (Equation 13): (12) (13) in, and These represent the preset minimum safe distances (including physical dimensions and control redundancy margin) between agents and between agents and static obstacles, respectively. These hard constraints will be transformed into strict feasible domain boundaries in subsequent steps through the control obstacle function.

[0036] Based on this, the present invention formalizes the hard constraints such as obstacle avoidance and collision prevention in the encirclement mission into a safety set. This set is defined as a barrier function. The superlevel set, namely: (14) in, This indicates that the system is in a safe state. This indicates that the system is located at the safety boundary. Up, and This means a collision has occurred. To ensure the system's safety, a control law needs to be designed so that for any condition satisfying... The initial state of the system, at any subsequent time. All meet Furthermore, we introduce the CBF criterion: given a continuously differentiable function... If an extended Class function (linear function) ), making it possible for all All have control input Satisfying the inequality: (15) Then it is called Let be a control barrier function for this system. All control inputs satisfying this inequality constitute the safety control set: (16) This inequality restricts the safety function. The decay rate: when the system approaches the safety boundary hour, Thus, the constraints force This generates a sufficient "repulsive force" to prevent the system from crossing the boundary, theoretically ensuring physical security.

[0037] (2) Positive eddy current artificial potential field; When an agent gets stuck in a deadlock in front of an obstacle, simple radial repulsion... Unable to provide the necessary tangential force to break the deadlock. Therefore, vortex potential force is introduced. To ensure the optimal escape trajectory, i.e., selecting the side with the shortest bypass path, the rotation direction of the vortex field must adaptively switch based on the relative geometric relationship between the target's gravitational direction and the obstacle's normal vector. Define the target gravitational direction vector. and the external normal vector of the obstacle Furthermore, a scalar form of the two-dimensional vector cross product is introduced, defining a dynamic rotation operator. : (17) Therefore, dynamic eddy current force is defined as the result of normal repulsive force mapped by an orthogonal rotation matrix, that is: (18) (19) in, This is the intensity coefficient of the eddy current potential force, used to adjust the amplitude of the tangential guiding effect. When When, the matrix performs a counter-clockwise rotation; when At this time, the matrix rotates clockwise. This mechanism ensures that the agent always moves around the obstacle along the tangent direction closer to the target, thus effectively escaping local deadlock.

[0038] (ii) PI-STEP-MADDPG algorithm; This invention addresses the challenges of low exploration efficiency due to sparse rewards and physical collision avoidance in complex dynamic environments during multi-agent trapping. It proposes a trapping algorithm, PI-STEP-MADDPG, based on positive eddy current potential fields and differentiable quadratic programming. This algorithm, built upon the STEP-MADDPG architecture, integrates PBRS potential energy reward shaping, VAPF local escape guidance, HOCBF safety constraints, and a DiffQP differentiable safety layer.

[0039] like Figure 1As shown, the algorithm consists of an Actor policy network, a PBRS reward shaping module, a VAPF prior eddy current module, a DiffQP differentiable safety layer, and a Critic value network. The Actor outputs the original action intent to represent the high-level encirclement strategy; PBRS constructs a differential reward based on a conservative potential field, providing global guidance for approaching the target, obstacle avoidance, and cooperative encirclement; VAPF, as a non-conservative physical prior embedded in the action output, injects tangential eddy current components in the local minimum region to break the deadlock caused by the collinear balance of attraction and repulsion forces; DiffQP performs secondary planning on the fused reference actions under the HOCBF collision avoidance hard constraint, outputting the final safe action; the Critic evaluates the value of the safe action and propagates the gradient back to the Actor via the differentiable safety layer, achieving joint optimization of the policy network and the safety layer.

[0040] In the action generation phase, this invention does not employ a hard switching mechanism, but instead constructs a state-dependent continuous fusion mechanism. The Actor first outputs the original nominal action, and the system assesses the deadlock risk based on the local potential gradient and obstacle distance: during normal cruise, the VAPF does not intervene, and the reference action degenerates into the Actor's original output; when entering the deadlock candidate region, the VAPF's takeover strength is continuously adjusted using the deadlock activation coefficient. To eliminate the conflict between the transition zone strategy action and the eddy current direction, an orthogonal projection mechanism is introduced before fusion to remove conflicting components from the original action, and then a reference input is generated through convex combination. Thus, the role of the VAPF in this invention is not to directly generate control quantities, but to collaborate with the Actor to construct reference actions, which are then uniformly submitted to DiffQP to solve for the final safe action under HOCBF constraints.

[0041] (1) Global guidance of potential function; The reward function directly determines the optimization direction of the reinforcement learning strategy. In the trapping task, giving sparse rewards only at the moment of success leads to difficulties in credit allocation and low exploration efficiency; while traditional artificial dense rewards can accelerate convergence, they are prone to destroying the optimal strategy of the original task, and may even cause reward hijacking due to the agent's pursuit of distance gains. To balance learning efficiency and policy invariance, this invention introduces a potential function-based reward shaping (PBRS) method, which encodes the trapping physical potential energy as a shaping reward: (20) in, It is a composite potential function, composed of the target's gravitational potential energy, the obstacle's repulsive potential energy, and the encirclement configuration potential energy. As a shaping reward superimposed with the original environmental reward The total reward function is expressed as: (twenty one) The shaping term used in this invention satisfies the standard potential difference form, and therefore belongs to a typical form of PBRS. The role of this type of reward shaping method is to provide additional guidance to the training process through the potential difference between adjacent states, without changing the optimization objective of the original trapping task. For the multi-agent trapping task of this invention, this means that the shaping reward is mainly used to enhance the learning signal and improve the exploration efficiency in the early stages of training, rather than changing the original task itself.

[0042] (1.1) Design of potential energy function; For the three main behaviors of target approach, obstacle avoidance and encirclement formation, the function is composed of three weighted components: target gravitational potential energy, obstacle repulsive potential energy and encirclement configuration potential energy.

[0043] (twenty two) in, and These represent the weighting coefficients for the target's gravitational potential energy, the obstacle's repulsive potential energy, and the encirclement configuration potential energy, respectively.

[0044] (a) Target gravitational potential energy; Traditional APF (Advanced Persistent Force) systems typically treat the target point as a gravitational source, causing the potential energy to increase with distance, and the negative gradient to generate an attractive force towards the target. However, in encirclement missions, the target is not collided with; ideally, it remains at a distance from the target. The desired encirclement is on the circumference. Therefore, this invention designs a potential trap type gravitational potential energy: (twenty three) in, This is the gravity gain coefficient. This function is applied when the agent is at a distance from the target. The potential energy reaches a maximum of 0 at a given distance, while it is negative at any location deviating from this distance. As the agent moves closer to the desired encirclement circle, the potential energy increases, resulting in a positive reward; conversely, moving away from the encirclement circle incurs a penalty. This design guides the agent to naturally converge and stably distribute itself around the target encirclement circle.

[0045] (b) Repulsive potential energy of the obstacle; The obstacle domain is defined as a low-potential-energy region, while the safe region is defined as a high-potential-energy region. In this case, the agent moving away from the obstacle will result in a potential energy difference. This corresponds to a positive reward. When the agent approaches an obstacle, a negative reward is generated. Specifically, the repulsive potential energy is defined as: (twenty four) in, For repulsive force gain, This represents the distance of the obstacle's influence. When the agent enters the obstacle's influence range and continues to approach, The potential energy decreases rapidly, generating a strong penalty signal that forces the agent to adjust its direction of movement. When the agent is outside the range of influence, its potential energy is reduced to zero to avoid unnecessary interference from distant obstacles on global guidance.

[0046] (c) Encirclement configuration potential energy; This is a key aspect demonstrating multi-agent cooperation. To achieve a uniform distribution of potential energy around the target during encirclement and pursuit, this invention correlates potential energy with the angular differences between adjacent agents, constructing an encirclement configuration potential energy: (25) Using Gaussian function form, Represents intelligent agents neighboring agents The angle difference relative to the target, when adjacent angle differences Approaching the expected interval At this point, the potential energy is at its maximum. This property guides the agent to make tangential movements on the circumference of the encirclement, actively adjusting its position to fill the gaps in the encirclement, thereby achieving a more uniform and stable cooperative encirclement configuration.

[0047] In summary, Equation (22) introduces explicit physical priors into the reward design of reinforcement learning through three types of potential energy terms. In the early stages of training, since the Critic network has not yet fully converged, the original value estimate is often noisy. However, the shaping reward based on the potential function can provide the agent with immediate and deterministic behavioral feedback. For example, when the agent moves towards the desired capture circle, it can immediately obtain a positive reward even if the capture has not yet been completed. This mechanism helps to significantly compress the ineffective exploration space and improve the sample utilization efficiency in the early stages of training. At the same time, since PBRS maintains policy invariance, its introduction will not change the optimal policy set of the original task.

[0048] (2) Safety action correction based on HOCBF-DiffQP; In (ii) (1), although PBRS provides global dense reward guidance, reward shaping alone is insufficient to guarantee strict safety in dense obstacle environments, especially the complex static obstacle avoidance and dynamic collision avoidance requirements in multi-agent collaboration. Furthermore, relying solely on the non-differentiable traditional QP layer for post-action correction leads to a long-term mismatch between policy output and actual actions, causing high-level policies to lose awareness of the underlying physical boundaries, thus triggering an adversarial relationship between the policy network and the safety mechanism. Therefore, this invention, based on a continuous-time control affine dynamics model, utilizes HOCBF to explicitly transform safety requirements into linear inequality constraints on the control input, embedding them into the DiffQP differentiable safety layer, thereby achieving end-to-end collaborative optimization of action safety correction and policy learning.

[0049] (2.1) Establishment of anti-collision constraints; In a multi-agent trapping system, the continuous physical states of the agents are denoted as: ,in and These represent position and velocity, respectively; the control input is the acceleration vector. Safety constraints are based on position vectors. Defined. Since the relative degree of position to acceleration is 2, the standard first-order CBF theory cannot be directly applied, and a higher-order safety constraint needs to be constructed using HOCBF. According to equations (13) and (14), the two types of obstacle functions are written in a single-term expansion form for the agent. Intelligent agents with neighbors Collision avoidance constraints between them are defined as follows: (26) For intelligent agents With the Avoidance constraints between static obstacles, defined as follows: (27) This indicates the preset minimum safe distance between intelligent agents. The preset minimum safe distance between the agent and the obstacle.

[0050] Collision avoidance inequality constraints: (28) The aforementioned collision avoidance inequality not only mathematically defines the safe zone, but each term also has an intuitive physical meaning. The left side of the inequality represents the control input, reflecting the underlying acceleration input. The direct effect on obstacle avoidance; the right side of the inequality is equivalent to a dynamic safety boundary, determined by the current state of the system, and specifically includes three parts: (1) Kinetic trend term This represents the tendency of an intelligent agent to continue moving due to inertia when there is no control intervention. (2) Approach speed penalty This reflects the relative approach speed between the agent and the obstacle. When approaching rapidly, it prompts the safety boundary to tighten dynamically, allowing for earlier braking or avoidance actions. (3) Distance buffer margin item This represents how much actual physical buffer space is between the agent's current position and the actual collision.

[0051] To incorporate constraints into the DiffQP solver, the coefficients from the above derivation are extracted, and a local constraint matrix is ​​defined. With boundary vector The collision avoidance conditions are organized as follows: (Regarding control input) Standard linear constraint form This process successfully simplified the originally complex nonlinear collision avoidance requirements into a linear constraint on acceleration, and the boundary of this constraint can be dynamically updated in real time with the environmental state, ultimately serving as the hard condition for the underlying optimization solution.

[0052] (29) (30) Furthermore, the derivation logic for dynamic cooperative collision avoidance among multiple agents is consistent with that for static obstacle avoidance. It only requires fixing the position coordinates of the obstacles. Replace velocity with the position coordinates of neighboring agents. With speed By considering the relative velocities of the two agents in the formula, it is easy to extend the cooperative collision avoidance constraints among multiple agents. Thus, the two types of hard safety constraints defined by geometric distance are both transformed into online solvable linear constraints with respect to the control input, and will serve as the unified input interface for the DiffQP differentiable safety layer in the next section.

[0053] (2.2) DiffQP Differentiable Safety Layer Design; In the previous section, this invention further transformed the defined geometric security constraints into linear inequality constraints with respect to the control input. This step solved the problem of mapping the security boundary in the location space to solvable input constraints of the execution layer. However, it did not resolve two other key contradictions: First, if the security layer is still regarded as a post-filter independent of the policy network, the actions output by the Actor network will be inconsistent with the actual actions executed in the environment for a long time, making it difficult for the high-level policy to accurately perceive the low-level security boundary. Second, traditional QP security layers usually cannot support gradient backpropagation. The value gradient of the Critic with respect to the final executed action cannot be backpropagated through the security layer to the Actor, thus creating a continuous adversarial relationship between the policy network and the low-level security mechanism. To address this, this invention further embeds HOCBF constraints into the DiffQP differentiable security layer, so that security correction is no longer a post-patch outside of training, but a component of the AC computation graph, thereby achieving joint optimization of the high-level policy and the low-level security mechanism.

[0054] 1) Robust safety QP design with slack variables; The original action output by the reinforcement learning Actor network is the nominal action. The core objective of a security filter is to find a method that, while satisfying the HOCBF physical constraints, is compatible with... The nearest control action In highly dynamic multi-agent trapping environments, systems often face extremely complex deadlock topologies (e.g., being cornered by multiple dynamic obstacles and teammates simultaneously). In such cases, if the system is required to strictly satisfy all physical boundaries, the intersection of multiple HOCBF-constrained linear polyhedra may be an empty set, causing the standard QP solver to fail.

[0055] In (ii) (1), the present invention constructs a potential energy function based on a conservative potential field. The invention employs PBRS reward shaping based on potential energy difference to maintain optimality. However, when the agent, obstacle center, and target point are collinear, the conservative potential field may exhibit a critical configuration with zero net force, leading to the agent getting trapped in local minima or deadlock. Therefore, this invention introduces a non-conservative vortex guidance field (VAPF). Since vortices contain curl components and cannot be represented as the gradient of the global scalar potential, they are not suitable as direct PBRS terms (otherwise, it would violate the theoretical premise of PBRS). Therefore, this invention places VAPF as a control reference in the Actor output layer instead of directly adding it to the reward; simultaneously, it maintains PBRS as a conservative potential energy term to ensure the invariance of reward shaping. This design utilizes vortices to break symmetry traps and improve exploration while avoiding the violation of optimality guarantees.

[0056] This invention defines the upper-layer intention as a reinforcement learning policy action. Dynamic combination with physical interventions in the eddy current field. To this end, a deadlock gradient threshold is defined. As a numerical tolerance for determining local minima, the influence radius of the obstacle is defined as... Introducing state-dependent deadlock activation coefficients. To strictly guarantee the continuity of action mapping at switching boundaries within a differentiable optimization framework (i.e., satisfying...) (Smoothing) avoids jitter in low-level instructions and computational anomalies in the implicit differentiation process, and... Designed as a product of distance smoothing weights and gradient smoothing weights, using a very small constant. Constructing smooth gradient norm To resolve the mathematical singularity problem where the vector magnitude and derivative are undefined at deadlock points, the global smoothing activation function is as follows: (31) in, Smoothing weights by distance With gradient smoothing weights The multiplication constructs two preconditions: whether the agent is close to an obstacle and whether it is close to a local minimum. Specifically, when the agent is far from the obstacle, the distance weights are in the form of a sigmoid. Approaching 0, the system maintains normal control primarily based on reinforcement learning strategies; when the agent enters the region affected by obstacles and the potential gradient gradually decreases, the cosine gradient weights... Gradually increase, thus A smooth ascent. This occurs only when the agent simultaneously satisfies the conditions of being close to an obstacle and the gradient being close to zero. A value close to 1 indicates that the system has entered a deadlock-high activation region, where eddy current guidance will gain stronger control. Conversely, if either condition is not met, the activation coefficient remains low, thus preventing premature intervention by physical priors in the normal cruise phase. Furthermore, a minimal constant is introduced into the gradient norm. This effectively avoids the problem of undefined vector magnitude derivatives at deadlock points, ensuring the smoothness of the switching process and the numerical stability of subsequent implicit differential solutions.

[0057] In traditional methods, deadlock gradient thresholding Radius of influence and distance scaling factor Hyperparameters are highly dependent on human experience; setting them too large can lead to an overly conservative system, while setting them too small makes it difficult to guarantee smooth transitions. Thanks to the DiffQP architecture used in this invention, every operation of the underlying control barrier and smoothing function supports implicit differentiation and gradient backpropagation. Therefore, it does not... , and Instead of being set as a static constant, the parameter is dynamically output by a lightweight multilayer perceptron based on the current local state, and jointly optimized in the loss function of the outer reinforcement learning layer. Through this parameter adaptation mechanism, the system can automatically learn more appropriate takeover timing and smoothness during training, thus transforming the traditional heuristic design that relies on manual parameter tuning into a learnable optimization process.

[0058] To overcome the system being in the transition region ( When reinforcement learning explores the phenomenon of control passivation caused by the opposite direction of action and physical field guidance, this paper introduces an orthogonal projection mechanism before action fusion. With eddy field Obtuse angle (i.e.) If the conflicting components are actively stripped away, the projection action is obtained. Based on the conflict-free actions after projection, the underlying fusion reference control commands are constructed through convex combination. , making when At that time, eddy field Able to acquire 100% control: (32) To ensure the robustness, control continuity, and numerical stability of the gradient of the solver, this invention introduces independent non-negative relaxation variable vectors into the objective function and constraints. ,in, , Let represent the set of currently active safety constraints. Using a precise penalty mechanism, hard constraints are transformed into soft constraints, leading to the following quadratic programming problem: (33) To ensure the safety of the solution and the numerical stability of implicit differentials, this invention assigns independent non-negative relaxation variables to each safety constraint. This is done to eliminate global conservatism in multi-obstacle scenarios and prevent the optimizer from artificially tightening safety boundaries by outputting negative values, thus avoiding action bias. In the objective function, a maximally linearly precise penalty is applied to the weights. The optimizer is forced to strictly maintain within the feasible region. This achieves zero tolerance for physical collision avoidance; simultaneously, it incorporates a small quadratic regularization coefficient. This strictly guarantees the strong convexity and positive definiteness of the Hessian matrix, thereby completely solving the problem of computational singularities in the failure of backpropagation to find the inverse under extreme constraint conflict conditions in DiffQP.

[0059] To address the issue of action manipulation caused by forced takeover at the underlying level in the Markov decision-making process of reinforcement learning, and to prevent cognitive bias in the policy network during training, a state-dependent takeover penalty term is introduced into the reward function of reinforcement learning. (34) in, This is used as a penalty weight. This mechanism means that when an agent is taken over by the physical field in a deadlock region, if it continues to output actions that significantly deviate from the correction direction of the safety layer, it will face additional penalties, thereby prompting the Actor network to gradually learn control behaviors that are coordinated with the underlying safety mechanism.

[0060] 2) Differentiability of KKT conditions and reconstruction of the loss function; The previous section constructed the HOCBF safety constraints as a quadratic programming (QP) problem with slack variables, and its optimal solution... It is the nominal action The function. To achieve autonomous learning, the error gradient of the final safe action calculated by the Critic network must be able to propagate back through this QP layer to the Actor network. However, the standard QP solution process involves... Operators in deep learning frameworks often struggle to support gradient backpropagation. This section employs a differentiable quadratic programming technique based on implicit function theorems, deriving the Karush-Kuhn-Tucker conditions at the optimal solution of the QP algorithm through implicit function differentiation. about The Jacobian matrix is ​​used to transform the security layer into a differentiable network layer. The derivation is divided into four steps: (a) The compact form of the QP problem; For ease of derivation, the QP problem is rewritten in the following compact form, defining the decision variables. ,in, Indicates control input, This represents the slack variable vector. To control the input dimensions, further define as well as in, Therefore, the original QP problem can be written as (35) (b) Implicit Jacobian matrix and gradient propagation; Traditional The operator is not differentiable. To enable the error gradient evaluated by the Critic network to propagate back to the Actor network, this invention employs a differentiable quadratic programming technique based on the implicit function theorem. Specifically, in the optimal solution... At this point, an implicit set of functional equations is constructed based on the Karush-Kuhn-Tucker (KKT) conditions of the aforementioned QP problem, and these equations are then applied with respect to the reference input. By taking the total derivative, the Jacobian matrix of the final safe action with respect to the reference action can be analytically solved: (36) In scenarios with dense obstacles, multiple activation constraints can lead to collinear gradients. This invention introduces slack variables and their penalty terms into the objective function and constraint design, which significantly improves the condition number of the KKT matrix, ensuring the validity of the Linear Independent Constraint Qualification (LICQ), thereby enhancing the numerical stability of Jacobian matrix solving and gradient backpropagation.

[0061] Obtain the Jacobian matrix Then, the security layer can be embedded at the output of the Actor network. Let the Actor network parameters be... Its output nominal action The final action is obtained after passing through the security layer. At this point, the Critic network no longer evaluates the original joint actions, but rather the value of the joint security actions. Therefore, the Actor's optimization objective can be reconstructed as: (37) According to the chain rule, the policy gradient is: (38) in, Obtained through backpropagation from the Critic network. , This is determined by the Actor network structure. This allows gradients to propagate back through a differentiable QP safety layer to the policy network, achieving joint optimization of the safety layer and the policy network.

[0062] It was obtained through backpropagation from the Critic network. This indicates that the DiffQP security layer has an implicit Jacobian with respect to the reference input, while This corresponds to the derivative of the projection and fusion module in equation (32). This allows the gradient to propagate back to the policy network through the differentiable QP security layer, achieving joint optimization of the security layer and the policy network. In essence, equation (38) introduces two intermediate mappings, a reference fusion layer and a differentiable security layer, on the basis of the deterministic policy gradient link, so that the higher-level policy can explicitly perceive the real effect of the lower-level security correction.

[0063] The algorithm flow for this chapter is as follows: (III) Simulation Experiments and Analysis; (1) Parameter settings; This section details the parameter settings related to the simulation environment, neural network architecture, reinforcement learning training, and physics field and security layer. It mainly includes basic parameters and specific parameters. The specific parameters used in the experiment are shown in Table 1, and the basic parameters are shown in Table 2.

[0064] Table 1 Among them, collision avoidance threshold and obstacle avoidance threshold are used to determine the safety boundary; maximum control input, maximum speed and sampling time step are used to limit the range of agent movement; learning rate, batch size, experience pool capacity, soft update coefficient and discount factor are used to control the training process; gravity, repulsion, cooperation and eddy current field gain are used to adjust the intensity of encirclement guidance, obstacle avoidance, cooperative control and deadlock escape; deadlock gradient threshold, obstacle influence radius and takeover penalty coefficient are used to characterize local minimum determination, physical field intervention range and policy deviation constraint degree, respectively; HOCBF related parameters are used to adjust the response speed of safety constraints and boundary approach strength.

[0065] Table 2 (2) Dynamic target encirclement experiment; In a dynamic target capture scenario, several randomly distributed static obstacles are placed within a 10x10 area. Three pursuers must cooperate to capture one static target without collision. The target possesses continuous maneuverability, requiring pursuers not only to avoid obstacles but also to adjust their collaborative capture strategy in real time based on the target's movement. This task places higher demands on the strategy's response speed, environmental adaptability, and execution layer security. The simulation results are analyzed below from four aspects: training convergence, collision behavior, the capture process, and execution layer security.

[0066] Figure 2 The total reward curves of four algorithms over 20,000 training rounds are shown. STEP's reward remained low and oscillated in dynamic environments for a long period, only beginning to slowly increase after approximately 14,000 rounds. PI-STEP-MADDPG, on the other hand, showed superior convergence speed and final reward level. In the later stages of training, the total reward of PI-STEP-MADDPG stabilized at approximately 210, significantly higher than Filter-PBRS-STEP's approximately 180, PBRS-STEP's approximately 150, and STEP's approximately 60. Furthermore, it exhibited the smallest variance, indicating that this method possesses better training stability and environmental adaptability in dynamic target acquisition scenarios.

[0067] Figure 3 The cumulative collision counts of each algorithm in a dynamic target encirclement scenario over 20,000 training rounds are presented. The collision counts of STEP and PBRS-STEP continuously increased during training, eventually reaching approximately 27,000 and 22,000 respectively, indicating that soft constraint mechanisms relying solely on reward / penalty or potential function rewards are insufficient to form stable obstacle avoidance strategies in complex dynamic environments. Filter-PBRS-STEP reduced the cumulative collision count to approximately 8,500, demonstrating that the safety filtering mechanism can suppress dangerous actions to some extent. However, due to the lack of differentiable connections between the safety layer and the policy network, conflicts still exist between high-level exploration and low-level safety correction. In contrast, PI-STEP-MADDPG had approximately 3,800 cumulative collision counts at the end of training, and the collision curve flattened significantly in the later stages, indicating that the differentiable safety layer continuously transmits safety constraint information to the policy network during training, enabling the policy to gradually learn control behaviors that meet safety boundary requirements.

[0068] Figure 4The cumulative number of collisions for each algorithm over 1000 test rounds was statistically analyzed. Random exploration noise was disabled during the testing phase to reflect the actual performance of the strategies under deterministic execution conditions. The results show that STEP and PBRS-STEP generated approximately 300 and 260 collisions, respectively, indicating that these two methods still struggle to effectively avoid dangerous maneuvers in dynamic environments. Filter-PBRS-STEP reduced the number of collisions to approximately 90, but due to its non-differentiable structure, it may still experience lag in action adjustment when facing sudden maneuvers from dynamic targets. PI-STEP-MADDPG generated only approximately 28 collisions in 1000 rounds, demonstrating that the differentiable safety layer not only improves the learning effect of safety constraints during the training phase but also significantly enhances the actual collision avoidance performance during the execution phase.

[0069] (3) Analysis of the collaborative mechanism of the differentiable security layer; To further analyze the mechanism of the differentiable quadratic programming safety layer in the policy learning process, this embodiment introduces two indicators in addition to task performance metrics: safety layer takeover rate and average action correction magnitude. A comparative analysis is then conducted between the traditional non-differentiable safety layer method (STEP-Filter) and the PI-STEP-MADDPG method proposed in this invention. These indicators can characterize the collaborative relationship between the safety layer and the policy network from the perspective of the training process, thereby reflecting the degree of internalization of safety constraints in policy learning. The safety layer takeover rate is defined as the proportion of time steps in which the safety layer significantly corrects the original action, i.e. in, To count the total number of time steps within the window, Indicates the number of agents in the system. This is an indicator function. It takes the value 1 when the condition in parentheses is true, and 0 otherwise. It is used to count the time steps when the security layer takes over. This serves as the threshold for determining takeover. This article uses... In other words, when the difference between the security layer output and the policy output exceeds this threshold, a security layer takeover is considered to have occurred at that moment. The average action correction amplitude is used to measure the average intervention intensity of the security layer on the policy output, and is defined as: The average action correction magnitude is defined as the average L2 norm of the difference between the safe action and the reference action, used to measure the average intervention intensity of the safety layer on the policy output. A lower takeover rate and a smaller action correction magnitude indicate that the policy network output is closer to the safe feasible region, and the posterior correction requirement of the safety layer is weaker. To ensure statistical robustness, this paper uses multiple random seeds to independently repeat the training for each experimental group, and calculates the mean and standard deviation of each indicator under the same number of training rounds. The curves in the figure represent the average results under multiple random seeds, and the shaded area represents the mean ± standard deviation.

[0070] In dynamic target encirclement scenarios, such as Figure 5 As shown in (a), both methods exhibited high security layer takeover rates in the early stages of training, with initial values ​​approaching 85%–90%, indicating that the policy had not yet formed a stable security behavior pattern in a dynamic environment, requiring frequent intervention and correction from the security layer. As training progressed, the takeover rate of PI-STEP-MADDPG decreased significantly more than that of STEP-Filter. At approximately 5000 rounds, the takeover rate of STEP-Filter remained around 50%, while that of PI-STEP-MADDPG had dropped to approximately 25%; at approximately 10000 rounds, the former remained around 30%, while the latter had dropped to 8%–10%; towards the end of training, PI-STEP-MADDPG approached 0%, while STEP-Filter maintained a level of approximately 18%–20%. This demonstrates that even in dynamic scenarios with more drastic state changes, differentiable security layers can effectively promote the internalization of security constraints into the policy network.

[0071] from Figure 5 As shown in (b), under dynamic conditions, the average action correction amplitude of PI-STEP-MADDPG is also significantly lower than that of STEP-Filter, and the convergence process is more stable. At approximately 5000 rounds, the average correction amplitude of STEP-Filter is about 0.63 m / s², while that of PI-STEP-MADDPG has decreased to about 0.30 m / s²; at approximately 10000 rounds, they are approximately 0.45 m / s² and 0.12 m / s², respectively; towards the end of training, PI-STEP-MADDPG further decreases to near zero, while STEP-Filter remains at approximately 0.28–0.30 m / s². This indicates that under conditions of continuous maneuvering of dynamic targets, PI-STEP-MADDPG can still learn control behaviors closer to the safe and feasible domain, thereby reducing the correction intensity of the safety layer on the original actions.

[0072] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A cooperative trapping method based on positive eddy current potential field and differentiable quadratic programming, characterized in that, Includes the following steps: Acquire local observation state information of each pursuit agent; The local observation state information is input into a pre-constructed policy network, and the policy network outputs the original action intentions of each pursuing agent. Based on the current state information and obstacle information of each pursuing agent, construct a high-order control obstacle function constraint; Based on the local potential field information of each pursuing agent, determine whether there is a risk of local deadlock. If so, fuse the tangential eddy current component of the positive eddy current artificial potential field with the original action intention to obtain a fused reference action; otherwise, use the original action intention as the reference action. The reference actions of each pursuing agent are input into a differentiable quadratic programming solver, and the safe actions of each pursuing agent are obtained and executed under the constraints of the higher-order control obstacle function. The state information of each pursuit agent after performing the safety action is obtained, and the shaping reward is calculated according to the reward shaping function based on the potential function. The environmental reward is obtained according to the environmental feedback. The shaping reward and the environmental reward are superimposed to obtain the total reward. Based on the state information of each pursuing agent after performing the safety action and the total reward, a value evaluation gradient is constructed. The value evaluation gradient is then backpropagated to the policy network through the differentiable quadratic programming solver to update the parameters of the policy network. The above steps are repeated until each pursuing agent meets the conditions for successful encirclement and capture, thus completing the cooperative encirclement and capture.

2. The cooperative trapping method based on positive eddy current potential field and differentiable quadratic programming according to claim 1, characterized in that, The process of constructing higher-order control barrier function constraints includes: An obstacle avoidance constraint function is defined based on the positional relationship between the pursuing agent and the obstacle, and an inter-agent collision avoidance constraint function is defined based on the positional relationship between the pursuing agent and adjacent pursuing agents. The first derivative of the obstacle avoidance constraint function is used to obtain the corresponding first-order auxiliary function, and the first derivative of the inter-agent collision avoidance constraint function is used to obtain the corresponding first-order auxiliary function. Differentiating the first-order auxiliary function corresponding to the obstacle avoidance constraint function yields a second-order expression for the control input of the pursuit agent; differentiating the first-order auxiliary function corresponding to the inter-agent collision avoidance constraint function yields a second-order expression for the control input of the pursuit agent. Based on the second-order expression, the constraint coefficient matrix and constraint boundary vector are extracted, and the obstacle avoidance constraint and the inter-agent collision avoidance constraint are transformed into linear inequality constraints with respect to the control input.

3. The cooperative trapping method based on positive eddy current potential field and differentiable quadratic programming according to claim 1, characterized in that, Determining whether there is a risk of local deadlock based on the local potential field information of each of the aforementioned pursuing agents includes: Calculate the resultant force vector of the target's gravitational direction vector and the obstacle's repulsive direction vector at the location of the pursuing agent; calculate the norm of the resultant force vector; when the pursuing agent enters the influence range of the obstacle and the norm of the resultant force vector is less than a preset gradient threshold, it is determined that there is a risk of local deadlock.

4. The cooperative trapping method based on positive eddy current potential field and differentiable quadratic programming according to claim 1, characterized in that, The process of obtaining the tangential eddy current component of the positive eddy current artificial potential field includes: Obtain the target gravitational direction vector and the obstacle outward normal vector at the location of the pursuing agent; calculate the cross product between the target gravitational direction vector and the obstacle outward normal vector, and determine the rotation direction of the vortex field according to the sign of the cross product; construct an orthogonal rotation matrix according to the rotation direction; map the obstacle outward normal vector through the orthogonal rotation matrix to obtain the tangential vortex component.

5. The cooperative trapping method based on positive eddy current potential field and differentiable quadratic programming according to claim 1, characterized in that, The tangential eddy current component of the positive eddy current artificial potential field is fused with the original action intent to obtain a fused reference action, including: The distance smoothing weight is calculated based on the distance between the pursuing agent and the obstacle, and the gradient smoothing weight is calculated based on the local potential field gradient norm. Multiplying the distance smoothing weight by the gradient smoothing weight yields the state-dependent deadlock activation coefficients. Determine whether the angle between the original intention of the action and the tangential eddy component is an obtuse angle. If so, peel off the conflicting component of the original intention of the action in the direction of the tangential eddy component to obtain the projected action. Based on the deadlock activation coefficient, the projection action and the tangential eddy current component are convexly combined to obtain the fused reference action.

6. The cooperative trapping method based on positive eddy current potential field and differentiable quadratic programming according to claim 1, characterized in that, The reference actions of each pursuing agent are input into a differentiable quadratic programming solver, and the safe actions of each pursuing agent are obtained under the constraints of the higher-order control obstacle function, including: The reference actions of each of the aforementioned pursuit agents are combined with relaxation variables to form decision variables, wherein the relaxation variables are non-negative variables; The objective function is constructed by the quadratic term of the deviation between the reference action and the control action to be solved, and the sum of the linear penalty term and the quadratic regularization term of the relaxation variable; The decision variables are solved under the constraints of the higher-order control barrier function to obtain the safe actions of each pursuit agent and the corresponding values ​​of the relaxation variables.

7. The cooperative trapping method based on positive eddy current potential field and differentiable quadratic programming according to claim 1, characterized in that, The process of constructing a reward shaping function based on a potential function includes: Construct a target gravitational potential energy function, which reaches its maximum value when the distance between the pursuing agent and the target is equal to the desired encirclement radius; Construct an obstacle repulsive potential energy function, wherein the obstacle repulsive potential energy function decreases as the distance decreases when the pursuing agent enters the influence range of the obstacle; Construct a trap configuration potential energy function, which reaches its maximum value when the angle difference between the pursuing agent and its neighboring pursuing agents relative to the target is equal to the desired angle interval; The composite potential energy function is obtained by weighted summing of the target gravitational potential energy function, the obstacle repulsive potential energy function, and the encirclement configuration potential energy function. The shaping reward is generated based on the difference between adjacent time steps of the composite potential function.

8. The cooperative trapping method based on positive eddy current potential field and differentiable quadratic programming according to claim 1, characterized in that, It also includes: adding a takeover penalty to the total reward, the takeover penalty being calculated based on the deviation between the reference action and the safe action.

9. The cooperative trapping method based on positive eddy current potential field and differentiable quadratic programming according to claim 1, characterized in that, The value assessment gradient is backpropagated to the policy network via the differentiable quadratic programming solver, including: Construct a set of implicit function equations with Karush-Kuhn-Tucker conditions at the optimal solution of the differentiable quadratic programming solver; Based on the implicit function equations, the Jacobian matrix of the decision variable relative to the reference action is solved using the implicit function theorem. Based on the Jacobian matrix, the value evaluation gradient is backpropagated from the security action to the reference action, and then backpropagated from the reference action to the policy network.

10. The cooperative trapping method based on positive eddy current potential field and differentiable quadratic programming according to claim 1, characterized in that, Updating the parameters of the policy network includes: Based on the status information of each of the pursuit agents after performing the safety action and the time difference error in the total reward calculation; Update the value network parameters based on the time difference error; The value assessment gradient is backpropagated to the policy network via the differentiable quadratic programming solver, and the policy network parameters are updated based on the backpropagated gradient. The target value network parameters and target policy network parameters are updated using a soft update method.

Images