Reinforcement learning-based three-dimensional space multi-agent elastic cooperative hunting control method and system

Abstract

The method and system for three-dimensional space multi-agent elastic cooperative hunting control based on reinforcement learning comprise the following steps: based on the established three-dimensional space agent dynamics model, an escape agent enhanced state Kalman filter prediction model is constructed, including extending the unknown input to the state vector of the escape agent to construct the enhanced state of the escape agent; the future state estimation of the escape agent is obtained from the enhanced state estimation value; a virtual spherical surface is constructed with the future position estimation of the escape agent as the center according to the future state estimation of the escape agent, and a dynamic hunting point is generated based on the tangential artificial potential field; the hunting agents are dynamically grouped based on the auction algorithm and reinforcement learning, and the matching between the hunting group and the hunting point is completed; an elastic cooperative hunting control law considering the actuator fault is constructed; a closed-loop multi-agent elastic cooperative hunting system is obtained according to the elastic cooperative hunting control law and the state of the hunting agent, and three-dimensional space elastic cooperative hunting of the escape agent is realized.

Description

Technical Field

[0001] This invention belongs to the fields of multi-agent cooperative control, reinforcement learning control, and fault-tolerant control, and particularly relates to a three-dimensional spatial multi-agent elastic cooperative trapping control method and system based on reinforcement learning. Background Technology

[0002] In recent years, multi-agent systems have been widely applied in fields such as UAV swarms, unmanned surface vessel swarms, unmanned underwater vehicle swarms, space robots, intelligent transportation, and target search and interception. Multi-agent cooperative encirclement control, as an important research direction in multi-agent systems, aims to achieve the tracking, encirclement, and capture of dynamically escaping targets through information interaction, task allocation, and cooperative control among multiple hunting agents.

[0003] In practical engineering applications, multi-agent cooperative trapping tasks typically face the following problems:

[0004] First, the trajectory of an escaping target is usually unknown, and its position, velocity, and attitude information may be affected by measurement noise, process disturbances, or communication delays. If the future state of the escaping target cannot be accurately predicted, the hunting agent is prone to lag in the capture process, leading to reduced capture efficiency or even capture failure.

[0005] Second, traditional artificial potential field methods typically employ fixed capture points or static enclosure structures, which are ill-suited to the dynamic movements of escaping targets. When an escaping target undergoes maneuvering changes, fixed capture points cannot be adjusted in real time according to the target's future state, thus affecting the uniformity of capture point distribution and the stability of coordinated capture.

[0006] Third, multiple hunting agents are usually distributed in a disordered manner in the initial state. If global coordinated control is carried out directly, it will lead to complex task allocation, large amount of computation, low training efficiency, and problems such as chaotic coordination within the group and uneven distribution of capture points.

[0007] Fourth, in practical unmanned systems, failures such as actuator malfunctions and decreased control efficiency are difficult to avoid. If the control law does not consider actuator failure compensation, the hunting agent may experience problems such as decreased control capability, increased tracking error, and disruption of intra-group consistency during the capture process.

[0008] While existing methods have achieved some success in multi-agent consensus control, reinforcement learning-based trapping control, and fault-tolerant control, they still have the following shortcomings: traditional control methods usually assume that the target state is known and lack a mechanism for predicting the future state of the escaping target; traditional artificial potential field methods mostly use fixed trapping points and lack the ability to dynamically schedule trapping points; independent reinforcement learning methods are prone to getting trapped in local optima and have difficulty handling large-scale dynamic grouping of agents and trapping point matching problems; some fault-tolerant control methods only consider the failure of a single actuator and fail to combine with dynamic grouping, trapping point matching, and reinforcement learning policy updates. Summary of the Invention

[0009] To address the problems in existing technologies, such as unknown escape target state, insufficient adaptability of fixed capture points, difficulty in disordered grouping of hunting agents, low training efficiency of reinforcement learning, and the impact of actuator failure on capture stability, this invention proposes a three-dimensional spatial multi-agent elastic cooperative capture control method and system based on reinforcement learning. This method can simultaneously achieve three-dimensional spatial multi-agent cooperative capture control that includes escape target future state prediction, dynamic generation of capture points, dynamic grouping and matching of hunting agents, and elastic compensation for actuator failure.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] The first aspect is a three-dimensional spatial multi-agent elastic cooperative trapping control method based on reinforcement learning, which includes the following steps:

[0012] S1. Establish a three-dimensional spatial intelligent agent dynamics model;

[0013] S2. Based on the three-dimensional spatial intelligent agent dynamics model, construct an enhanced state Kalman filter prediction model for the escape agent, including extending the unknown input to the state vector of the escape agent to construct the enhanced state of the escape agent; and obtaining the future state estimate of the escape agent from the enhanced state estimate.

[0014] S3. Based on the future state estimate of the escaping agent, construct a virtual sphere centered on its future position estimate, and generate dynamic capture points based on the tangential artificial potential field.

[0015] S4. Dynamically group hunting agents based on auction algorithms and reinforcement learning, and complete the matching between hunting groups and hunting points;

[0016] S5. Construct a flexible cooperative containment control law that takes into account actuator failures based on the matching results;

[0017] S6. Based on the elastic collaborative capture control law and the state of the hunting agent, a closed-loop multi-agent elastic collaborative capture system is obtained to realize the three-dimensional spatial elastic collaborative capture of the escaped agent.

[0018] Preferably, S2 includes:

[0019] Considering process noise and observation noise, the escape agent model is rewritten; the rewritten escape agent model satisfies the unknown bounded constraints; the unknown input is extended to the state vector of the escape agent model using auxiliary variables describing the changing trend of the unknown input, thus constructing the enhanced state of the escape agent model; a continuous Kalman filter is designed based on the enhanced state of the escape agent model.

[0020] Preferably, S3 includes:

[0021] Extract the future position estimate of the escaping agent from the future state estimate. Construct a unit virtual sphere with the future position estimate as the center, and generate a set of capture points on the unit virtual sphere. For any two capture points, construct a distance-related repulsive force, and calculate the force acting on the first capture point based on the set of projection points of the capture points on the virtual sphere. The resultant force of each capture point; the resultant force is tangentially decomposed to obtain the tangential force; the velocity and position of the capture points are updated based on the tangential force.

[0022] Preferably, S4 includes:

[0023] S41. Divide the hunting agent into... There are several encirclement groups, each corresponding to one encirclement point; for the first... The hunting agent and the first Calculate the distance between two points of encirclement;

[0024] S42. Define the instantaneous value function of the hunting agent for the capture point based on the distance; define the net value function based on the global price and instantaneous value function of the capture point; determine the optimal and suboptimal capture points based on the net value function.

[0025] S43. Based on the net value function, use the auction algorithm to update the local and global prices of the catch point.

[0026] Preferably, S4 also includes constructing a comprehensive reward function for training the reinforcement learning policy network:

[0027] in, To match rewards for the auction, As a reward for consistency within the group, A reward will be given for capturing the target. Penalties for unreasonable pricing To balance the grouping, and These are the weighting coefficients.

[0028] As a preferred embodiment, S5 includes:

[0029] S51, Regarding the first The first of the encirclement teams A hunting agent is defined as follows: its tracking error relative to the optimal capture point is defined as the difference between the estimated position of the hunting agent and the optimal capture point; the intra-group consistency error of the capture group is defined as the difference between the estimated positions of the hunting agents within the group; the coupling feedback function is defined as a nonlinear function describing the coupling relationship between the hunting agent and the optimal capture point.

[0030] S52. Set a fault factor that takes into account the failure of the actuator; design an elastic collaborative containment control law based on the tracking error feedback gain, coupling feedback gain, intra-group consistency feedback gain and fault factor.

[0031] As a preferred option, in S52:

[0032] The fault factors are set as follows:

[0033]

[0034] And satisfy:

[0035]

[0036] when When the actuator is working normally, it is in good working order; when At that time, the actuator partially failed;

[0037] The design law for the flexible and collaborative encirclement control is as follows:

[0038]

[0039] in, This is an estimate of the actuator failure factor. , , These are the error feedback gain, coupling feedback gain, and intra-group consistency feedback gain, respectively; the error feedback gain, coupling feedback gain, and intra-group consistency feedback gain are updated based on the total reinforcement learning reward.

[0040] As a preferred option, S6 also includes a design group consensus feedback gain update mechanism:

[0041] Based on the relationship between the intra-group consistency error and the preset intra-group consistency error threshold, the action state of the intra-group consistency feedback gain is adjusted; when the intra-group consistency error does not exceed the preset threshold, the intra-group consistency feedback gain is kept updated stably according to the total reinforcement learning reward; otherwise, the intra-group consistency feedback gain is updated through the total reinforcement learning reward to enhance the effect of intra-group consistency feedback.

[0042] Preferably, S6 also includes providing the stability conditions for the closed-loop system:

[0043] Construct the following Lyapunov function:

[0044]

[0045] in, , , It is a symmetric positive definite matrix. and Let be the weighting coefficients, and satisfy:

[0046]

[0047] The first term of the Lyapunov function describes the tracking error energy of the hunting agent towards the optimal capture point, the second term describes the coupling feedback error energy, and the third term describes the intra-group consistency error energy. By appropriately selecting the control gain... , , and positive definite matrix , , Such that the derivative of the Lyapunov function satisfies:

[0048]

[0049] The tracking error at the capture point, the coupling feedback error, and the intragroup consistency error all gradually converge.

[0050] Secondly, a three-dimensional spatial multi-agent elastic cooperative trapping control system based on reinforcement learning includes:

[0051] The dynamics modeling module is used to build dynamic models of intelligent agents in three-dimensional space.

[0052] The state prediction module is used to construct an enhanced state Kalman filter prediction model for an escape agent based on a three-dimensional spatial agent dynamics model. This includes extending the unknown input into the state vector of the escape agent to construct the enhanced state of the escape agent.

[0053] The dynamic capture point generation module is used to extract future position estimates based on the future state estimates of the escaping agent, construct a virtual sphere centered on the future position estimates, and generate dynamic capture points based on the tangential artificial potential field.

[0054] The dynamic grouping and matching module is used to dynamically group hunting agents based on auction algorithms and reinforcement learning, and to match the hunting groups with the hunting points.

[0055] The cooperative control law construction module is used to construct a flexible cooperative containment control law that takes into account actuator failures based on the matching results.

[0056] The closed-loop encirclement and capture execution module is used to obtain a closed-loop multi-agent elastic cooperative encirclement and capture system based on the elastic cooperative encirclement and capture control law and the state of the hunting agent, so as to realize the three-dimensional spatial elastic cooperative encirclement and capture of the escape agent.

[0057] The reinforcement learning-based three-dimensional spatial multi-agent elastic cooperative trapping control system is used to implement the reinforcement learning-based three-dimensional spatial multi-agent elastic cooperative trapping control method and its steps as described in the first aspect.

[0058] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0059] 1. Unlike traditional capture and control methods that typically rely on the current state of the escaping target or assume its location is known, this invention employs a combined approach of enhanced state Kalman filtering and joint estimation of the escaping target's unknown inputs. This expands the unknown inputs of the escaping agent into enhanced states, and predicts the escaping agent's future position, velocity, and attitude based on current observations. Consequently, subsequent capture point generation, dynamic grouping, and control law design are all based on the escaping target's future state, reducing capture lag and improving the foresight and target state estimation accuracy of three-dimensional collaborative capture.

[0060] 2. Unlike traditional artificial potential field methods that employ fixed capture points or static enclosure structures, this invention uses a dynamic capture point generation scheme that combines "prediction of the escape target's future state, virtual sphere construction, and tangential artificial potential field." By constructing a virtual sphere centered on the predicted escape target location, and dynamically adjusting the capture point positions using repulsive forces, radial forces, and tangential forces between capture points, multiple capture points can be uniformly distributed around the estimated future location of the escape target in real time. This avoids the problem of poor adaptability of fixed capture points to target maneuverability changes, improving the uniformity of capture point distribution and the stability of dynamic capture.

[0061] 3. Unlike traditional multi-agent trapping methods that rely on direct global consistency control, static task allocation, or simply reinforcement learning strategy training, this invention employs a dynamic grouping and matching scheme combining auction algorithms and reinforcement learning. By constructing a distance-value function between the hunting agent and the trapping point, and combining matching rewards, trapping arrival rewards, intra-group consistency rewards, bidding rationality penalties, and group equilibrium penalties, the auction algorithm is used to achieve optimal matching between the grouping of hunting agents and between the trapping group and the trapping point. This reduces the computational complexity of unordered hunting agent grouping and matching, improves the efficiency of dynamic task allocation and strategy training convergence, and enables the hunting agents to complete the three-dimensional trapping task in a "first intra-group consistency, then collaborative trapping" manner.

[0062] 4. Unlike existing methods where state prediction, capture point generation, agent grouping, and fault-tolerant control are designed independently, this invention integrates enhanced state Kalman filtering prediction, tangential artificial potential field capture point scheduling, auction reinforcement learning group matching, and actuator failure compensation control law into a unified closed-loop control framework. The above technical features form a sequentially related data flow: the future state prediction result of the escaping target is used to generate dynamic capture points, which serve as targets for auction reinforcement learning group matching. The group matching result is further used to construct the elastic collaborative capture control input. This allows the capture point tracking capability and intra-group consistency convergence capability to be maintained even when the hunter agent's actuator experiences partial failure, while simultaneously improving target prediction accuracy, capture point adaptability, group matching efficiency, capture success rate, and system fault-tolerant stability. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the method flow of Embodiment 1 of the present invention;

[0064] Figure 2 This is a schematic diagram of a three-dimensional multi-agent cooperative encirclement scenario according to Embodiment 1 of the present invention;

[0065] Figure 3 This is a schematic diagram of the dynamic model of the unmanned aerial vehicle system in Embodiment 1 of the present invention;

[0066] Figure 4 This is a schematic diagram of the topology of the multi-agent system in Embodiment 1 of the present invention;

[0067] Figure 5 This is a graph showing the position estimation error of the escaped agent in Embodiment 1 of the present invention.

[0068] Figure 6 This is a graph showing the velocity estimation error of the escape agent in Embodiment 1 of the present invention.

[0069] Figure 7 This is a graph showing the attitude estimation error of the escaped agent in Embodiment 1 of the present invention.

[0070] Figure 8 This is a comparison chart of the estimation performance of the continuous Kalman filtering method in Embodiment 1 of the present invention and the traditional discrete Kalman filtering method;

[0071] Figure 9 This is a comparison chart of the average iteration time of different capture methods in Embodiment 1 of the present invention;

[0072] Figure 10 This is a comparison chart of the success rates of different capture methods in Embodiment 1 of the present invention;

[0073] Figure 11 This is a diagram showing the result of multi-agent elastic cooperative trapping based on reinforcement learning in Embodiment 1 of the present invention. Detailed Implementation

[0074] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0075] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0076] This paper focuses on a collaborative maneuvering mission involving a swarm of unmanned aerial vehicles (UAVs) in three-dimensional space. The system comprises one escape agent and multiple hunting agents. The escape agent maneuvers in three-dimensional space, while the hunting agents coordinate to capture the escape agent through information exchange, state estimation, dynamic grouping, and flexible control.

[0077] This invention first establishes a dynamic model of the escaping agent and the hunting agent in three-dimensional space. Second, it extends the unknown input of the escaping agent into an enhanced state, designs an enhanced state Kalman filter, and realizes the prediction of the escaping agent's future position, velocity, and attitude. Then, it constructs a virtual sphere centered on the estimated future position of the escaping agent, and generates uniformly distributed dynamic capture points in real time based on a tangential artificial potential field. Further, it uses an auction algorithm and reinforcement learning method to dynamically group the disordered hunting agents and achieve matching between capture groups and capture points. Finally, it constructs an elastic cooperative capture control law that considers partial actuator failure, enabling the hunting agents to first achieve consistency within their groups and then synchronously approach the corresponding capture points, thereby completing the three-dimensional cooperative capture of the escaping agent.

[0078] The inventive aspect of this invention lies in its approach to multi-agent cooperative encirclement scenarios in three-dimensional space. It incorporates the unknown inputs of escaping agents into the enhanced state, predicts the future multi-step states of the escaping target through continuous Kalman filtering, and further utilizes the prediction results for virtual spherical encirclement point generation, auction reinforcement learning dynamic grouping and matching, and actuator fault compensation control law design. This forms an integrated closed-loop technical solution encompassing "target prediction—encirclement point scheduling—grouping and matching—elastic control." This combination reduces encirclement lag and improves the adaptability of dynamic encirclement, grouping efficiency, and encirclement stability under fault conditions.

[0079] Example 1:

[0080] like Figure 1 The illustrated method for elastic cooperative trapping control of multiple agents in three-dimensional space based on reinforcement learning includes the following steps:

[0081] S1. Establish a three-dimensional spatial intelligent agent dynamic model.

[0082] Let the first The state variables of each agent are:

[0083]

[0084] in, , , These represent the position coordinates of the agent in three-dimensional space. Indicates the speed of the intelligent agent. Indicates pitch angle, Indicates the yaw angle. When When, it indicates an escaping agent; when "At that time" indicates the hunting of intelligent agents.

[0085] The kinematic model of an intelligent agent in three-dimensional space is as follows:

[0086]

[0087] in, For the control input of the intelligent agent.

[0088] The state equation of the escaping agent is:

[0089]

[0090] in, For the unknown bounded input of the escaping agent, For the observation output of the escaping agent, , , It is a system matrix with appropriate dimensions.

[0091] The state equation of the hunting agent is:

[0092]

[0093] in, For the first The state matrix of a hunting agent The input for the flexible collaborative containment control to be designed.

[0094] S2. Construct an enhanced state Kalman filter prediction model for the escaped agent.

[0095] Considering process noise and observation noise The escape agent model is rewritten as follows:

[0096]

[0097] in, For process noise, To observe the noise and satisfy the unknown bounded constraints:

[0098]

[0099] in, and These are the shape matrices for process noise and observation noise, respectively.

[0100] To simultaneously estimate the state and unknown input of the escaping agent, the unknown input is extended into the state vector to construct an enhanced state:

[0101]

[0102] in, An auxiliary variable used to describe the changing trend of unknown inputs.

[0103] The enhanced state representation of an escape agent is as follows:

[0104]

[0105] Design a continuous Kalman filter based on the above formula:

[0106]

[0107] in, To enhance the state estimate, To enhance the state estimation error covariance matrix, To enhance the Kalman filter gain, Predict the step size for the future.

[0108] The future state estimate, unknown input estimate, and future observation prediction of the escaping agent are obtained from the enhanced state estimate:

[0109]

[0110] in, This indicates the extraction of a specified consecutive element from the enhanced state vector.

[0111] S3. Dynamic capture point generation based on tangential artificial potential field

[0112] Future position estimation describes the positional components extracted from future state estimates and used to construct the virtual sphere and generate capture points. This is used in the context of future position estimation for escaped agents. Construct a unit virtual sphere with the center of the sphere as the center, and generate a set of capture points on this virtual sphere:

[0113]

[0114] in, Indicates the first One encirclement point, =1,2,…, , This refers to the number of capture points.

[0115] No. Each capture point is represented as follows:

[0116]

[0117] in, For estimating the future position of the escaping agent, To determine the future position of the escaped agent pointing to the first The unit direction vector of each capture point.

[0118] For any two encirclement points and Construct distance-dependent repulsive forces:

[0119]

[0120] in,

[0121]

[0122] Calculations applied to the first The combined force of the various encirclement points:

[0123]

[0124] in, This is the set of projection points of the capture point onto the virtual sphere.

[0125] To ensure the capture points move tangentially along the spherical surface and remain uniformly distributed, the resultant force is decomposed tangentially to obtain the tangential force:

[0126]

[0127] Update the velocity and position of the capture point based on the tangential force:

[0128]

[0129] in, The step size constant is updated for velocity. Using the above method, multiple capture points can be dynamically and uniformly distributed in real time around the estimated future location of the escaping agent.

[0130] S4. Dynamic grouping and capture point matching based on auction algorithm and reinforcement learning.

[0131] S41. Suppose the hunting agent is divided into... There are several encirclement groups, each corresponding to a single encirclement point. For the first... The hunting agent and the first There are several encirclement points, and the distance between them is defined as:

[0132]

[0133] S42, Definition of the The hunting agent targets the first... Instant value function and net value function of each capture point:

[0134]

[0135] in, It is a positive constant greater than the maximum possible distance. For the first The overall price of each capture point.

[0136] Determine the optimal and suboptimal capture points based on the net value function:

[0137]

[0138] S43. Update the local and global prices according to the auction algorithm:

[0139]

[0140] in, For the local price update step size constant, This represents the number of hunting agents within the encirclement group.

[0141] S44. Construct a comprehensive reward function for training the reinforcement learning policy network:

[0142] in, To match rewards for the auction, As a reward for consistency within the group, A reward will be given for capturing the target. Penalties for reasonable pricing To balance the grouping, and These are the weighting coefficients.

[0143] By using the matching results of the auction algorithm to assist the reinforcement learning policy network update, the hunting agent can obtain reasonable grouping strategies, capture point matching strategies, and cooperative capture actions, thus avoiding the problem that the computational complexity of traditional methods increases rapidly with the number of agents.

[0144] S5. Construct a flexible cooperative containment control law that considers actuator failure.

[0145] S51, Regarding the first The first of the encirclement teams A hunting agent is defined relative to the optimal trapping point. The tracking error is:

[0146]

[0147] Define the intragroup consistency error as:

[0148]

[0149] in, This indicates that within the same capture group, excluding the first... Other hunting agents besides the hunting agent.

[0150] Define the coupling feedback function as follows:

[0151]

[0152] in, This is a nonlinear function describing the coupling relationship between the hunting agent and the optimal capture point.

[0153] S52. Considering partial actuator failure, let the actuator failure factor be:

[0154]

[0155] And satisfy:

[0156]

[0157] when When the actuator is working normally, it is in good working order; when At that time, the actuator partially failed.

[0158] The design law for the flexible and collaborative encirclement control is as follows:

[0159]

[0160] in, This is an estimate of the actuator failure factor. , , These are the error feedback gain, coupling feedback gain, and intragroup consistency feedback gain, respectively.

[0161] The control gains, including error feedback gain, coupling feedback gain, and intra-group consistency feedback gain, are updated based on the total reinforcement learning reward (comprehensive reward function):

[0162]

[0163] in, For gain learning rate, To enhance the overall learning reward.

[0164] S6. Establish a closed-loop multi-agent elastic collaborative capture system

[0165] S61. Substituting the expression of the elastic cooperative trapping control law into the state equation of the hunting agent, we obtain the closed-loop system:

[0166]

[0167] Through a closed-loop multi-agent elastic collaborative encirclement system, the hunting agents first achieve state consistency within the same encirclement group during the encirclement process, and then approach the corresponding encirclement point as a whole, ultimately achieving three-dimensional spatial elastic collaborative encirclement of the escaped agents.

[0168] S62. Design an intra-group consensus feedback gain adjustment mechanism and provide the stability conditions for the closed-loop system.

[0169] The intra-group consensus feedback gain adjustment mechanism describes how the strength of the intra-group consensus feedback gain is adjusted based on the intra-group consensus error threshold during closed-loop operation, and provides stability constraints through the Lyapunov function. After the closed-loop multi-agent elastic cooperative trapping system is established, in order to ensure that the hunting agents can operate in the manner of "first intra-group consensus, then cooperative trapping", the intra-group consensus feedback gain needs to be adjusted according to the magnitude of the intra-group consensus error.

[0170] When the intra-group consistency error satisfies When the state difference between hunting agents within the same hunting group is small, it indicates that the state difference between them is already small. At this point, maintaining the consistency feedback gain within the group is sufficient. Updates are made according to the total reward of reinforcement learning to avoid excessive consistency feedback within the group causing system oscillations and to maintain stability within the group.

[0171] When the intra-group consistency error satisfies This indicates that there are still significant state differences among hunting agents within the same hunting group. In this case, the total reward from reinforcement learning can be used to improve the consistency feedback within the group. Updates are performed to enhance intra-group consistency feedback and improve intra-group collaborative convergence speed. Among these updates, This is the preset intra-group consistency error threshold.

[0172] To further prove the stability of the closed-loop system, the following Lyapunov function is constructed:

[0173]

[0174] in, , , It is a symmetric positive definite matrix. and Let be the weighting coefficients, and satisfy:

[0175]

[0176] The first term in the Lyapunov function describes the tracking error energy of the hunting agent towards the optimal capture point, the second term describes the coupling feedback error energy, and the third term describes the intra-group consistency error energy. By appropriately selecting the control gain... , , and positive definite matrix , , Such that the derivative of the Lyapunov function satisfies:

[0177]

[0178] Therefore, it can be seen that the tracking error of the capture point, the coupling feedback error, and the intra-group consistency error in the closed-loop system can all gradually converge, thus ensuring that the multi-agent system can still achieve stable three-dimensional spatial elastic cooperative capture control even when the actuator part fails.

[0179] Example 2:

[0180] A reinforcement learning-based method for elastic cooperative trapping control of multiple agents in three-dimensional space includes the following steps:

[0181] Step (1) System Initialization

[0182] Initialize the escape agent state Hunting agent state Number of encirclement points Number of encirclement teams Auction algorithm parameters , Tangential potential field step size Strengthen the weight of learning rewards , Control gain , , and prediction step size .

[0183] according to Figure 2 The encirclement scenario shown depicts an escape agent and multiple hunting agents deployed in a three-dimensional space; according to... Figure 3 The dynamic model of the unmanned aerial vehicle is shown, and the state equation of the intelligent agent is established.

[0184] Step (2) Predicting the future state of the escaping agent

[0185] Based on the current observation output of the escaping agent Construct an enhanced state Kalman filter to obtain the escape agent at future time steps. State estimate Unknown input estimates and observed predicted values .

[0186] This step allows the hunting agent to no longer rely solely on the current location of the escaped agent, but instead generate capture points and plan paths based on the future state of the escaped agent, thereby improving the foresight of the capture mission.

[0187] Step (3) Generation of dynamic capture points

[0188] by Construct a unit virtual sphere with the center of the sphere, and initialize multiple capture points on the sphere. Calculate the repulsive and tangential forces between the capture points, and update the velocities and positions of the capture points to ensure that the multiple capture points are uniformly distributed around the escaping agent.

[0189] Compared with traditional fixed capture point methods, the capture point in this paper can move dynamically with the future estimated location of the escaping agent, making it suitable for capture tasks where the trajectory of the escaping agent is unknown and its maneuverability is changing.

[0190] Step (4) Dynamic grouping and matching of hunting agents

[0191] Calculate the instantaneous value function and net value function based on the distance between the hunting agent and the capture point. Determine the optimal and suboptimal capture points for each hunting agent and perform price iteration.

[0192] During the price iteration process, a mechanism of "target price - price iteration - optimal allocation" is adopted to enable the hunting agent to automatically form a hunting group and complete the matching between the hunting group and the hunting point.

[0193] Furthermore, a reinforcement learning reward function is constructed, and the policy network is trained using the PPO reinforcement learning algorithm, enabling the hunting agent to achieve a balance between encirclement point matching, intra-group consistency, encirclement arrival, and bid rationality.

[0194] Step (5) Flexible Cooperative Encirclement and Control

[0195] For each encirclement group, calculate the tracking error of the hunting agent relative to the optimal encirclement point, calculate the intra-group consistency error, and calculate the coupling feedback term.

[0196] Considering partial actuator failure, an actuator failure model is established, and a flexible collaborative containment control law is constructed.

[0197] During the encirclement process, when the intra-group consistency error is large, the role of the intra-group consistency control term is enhanced, so that the hunting agents in the same encirclement group can quickly gather together; when the intra-group consistency error meets the preset threshold, the intra-group consistency feedback term is kept stable to avoid excessive feedback leading to system oscillation, so that the encirclement group remains stable and approaches the corresponding encirclement point as a whole.

[0198] Step (6) Closed-loop system operation and judgment of the completion of the capture

[0199] Substituting the elastic cooperative trapping control law into the dynamic equations of the hunting agent, a closed-loop system is obtained. During the operation of the closed-loop system, the following loop is continuously executed:

[0200] First, real-time observation information of the escaping agent is collected;

[0201] Secondly, the future state of the escaping agent is predicted by enhancing the state Kalman filter;

[0202] Secondly, the capture point is updated based on the tangential artificial potential field;

[0203] Then, dynamic grouping and capture point matching of hunting agents are performed based on auction algorithms and reinforcement learning strategies.

[0204] Finally, the elastic cooperative encirclement control law is used to drive the hunting agent to move towards the corresponding encirclement point.

[0205] When all the capture groups have reached their respective capture points and the escaping agents are within the preset capture area, the capture task is considered complete; otherwise, the state prediction step is returned to continue iterating.

[0206] Simulation example:

[0207] To verify the effectiveness of the method described in this invention, a three-dimensional multi-agent cooperative encirclement simulation environment was constructed. The simulation objects include one escaping agent and multiple hunting agents. The escaping agent moves in three-dimensional space according to a preset maneuver trajectory, and the hunting agents, based on the elastic cooperative encirclement control method proposed in this invention, complete the cooperative encirclement of the escaping agent.

[0208] In this simulation example, the positions, velocities, attitude angles, communication topology, and control algorithm parameters of the escaping agent and the hunting agent are first initialized. Then, the state equations of the escaping agent and the hunting agent are established based on the three-dimensional agent dynamics model. Figure 4 The topology shown establishes communication relationships within a multi-agent system, ultimately yielding the control law gain:

[0209]

[0210] At each simulation time, firstly, the future state of the escaping agent is predicted based on its current observation information using an enhanced state Kalman filter. Secondly, a virtual sphere is constructed centered on the predicted future position of the escaping agent, and uniformly distributed capture points are dynamically generated based on the tangential artificial potential field. Then, based on the distance value function between the hunting agent and the capture point, combined with the auction algorithm and reinforcement learning reward function, the hunting agents are dynamically grouped, and the matching between the capture groups and the capture points is completed. Finally, the tracking error of the capture point, the intra-group consistency error, and the coupling feedback term are calculated based on the matching results, and the control input of each hunting agent is output using an elastic cooperative capture control law containing actuator fault compensation terms.

[0211] To verify the control performance of this invention under actuator failure conditions, some hunting agents were subjected to partial actuator failure during the simulation, resulting in a decrease in their control efficiency. During the simulation, the escape agent's state estimation error, the distribution of capture points, the grouping results of the hunting agents, the intra-group consistency error, the capture success rate, and the algorithm iteration time were recorded.

[0212] Simulation results show that the position estimation error, velocity estimation error, and attitude estimation error of the escaping agent can all converge and remain stable over time. Figure 5 This indicates the position estimation error. Figure 6 This indicates the speed estimation error. Figure 7 This represents the attitude estimation error.

[0213] Figure 8 The continuous Kalman filtering prediction method described in this invention has a faster convergence speed and higher estimation accuracy compared to the traditional discrete Kalman filtering method. Figure 9 and Figure 10The results show that, compared with traditional artificial potential field methods and ungrouped consistency control methods, the method described in this invention can reduce the average iteration time and improve the success rate of encirclement. Figure 11 The final result shows the successful capture of an escaped agent by multiple hunting agents in three-dimensional space.

[0214] Simulation results show that even in situations such as unknown escape target trajectories, disordered distribution of hunting agents, and partial failure of actuators, the method described in this invention can still achieve stable and efficient three-dimensional space multi-agent elastic cooperative encirclement control.

[0215] In summary, the present invention proposes a three-dimensional space multi-agent elastic cooperative capture control method based on reinforcement learning. It predicts the future state of escaping agents by enhancing state Kalman filtering, dynamically generates capture points by tangential artificial potential fields, realizes dynamic grouping of hunting agents and matching of capture points by auction algorithm and reinforcement learning, and achieves elastic cooperative capture by actuator fault compensation control law. This method can effectively improve the capture success rate, real-time performance and fault tolerance of multi-agent systems in complex three-dimensional space.

[0216] Example 3:

[0217] A reinforcement learning-based three-dimensional multi-agent elastic cooperative trapping control system includes:

[0218] The dynamics modeling module is used to build dynamic models of intelligent agents in three-dimensional space.

[0219] The state prediction module is used to construct an enhanced state Kalman filter prediction model for an escape agent based on a three-dimensional spatial agent dynamics model. This includes extending the unknown input into the state vector of the escape agent to construct the enhanced state of the escape agent.

[0220] The dynamic capture point generation module is used to extract future position estimates based on the future state estimates of the escaping agent, construct a virtual sphere centered on the future position estimates, and generate dynamic capture points based on the tangential artificial potential field.

[0221] The dynamic grouping and matching module is used to dynamically group hunting agents based on auction algorithms and reinforcement learning, and to match the hunting groups with the hunting points.

[0222] The cooperative control law construction module is used to construct a flexible cooperative containment control law that takes into account actuator failures based on the matching results.

[0223] The closed-loop capture execution module is used to obtain a closed-loop multi-agent elastic cooperative capture system based on the elastic cooperative capture control law and the state of the hunting agent, so as to realize the three-dimensional spatial elastic cooperative capture of the escape agent.

Claims

1. A three-dimensional spatial multi-agent elastic cooperative trapping control method based on reinforcement learning, characterized in that, Taking a collaborative maneuvering mission involving a swarm of drones in three-dimensional space as an example, the system comprises one escape agent and multiple hunting agents. The escape agent maneuvers in three-dimensional space, and the hunting agents achieve collaborative maneuvering against the escape agent through information exchange, state estimation, dynamic grouping, and elastic control. The system includes the following steps: S1. Establish a three-dimensional spatial intelligent agent dynamics model; S2. Based on the three-dimensional spatial intelligent agent dynamics model, construct an enhanced state Kalman filter prediction model for the escape agent, including extending the unknown input to the state vector of the escape agent to construct the enhanced state of the escape agent; obtain the future state estimate of the escape agent from the enhanced state estimate to achieve the prediction of the future position, velocity and attitude of the escape agent; S3. Based on the future state estimate of the escaping agent, construct a virtual sphere centered on its future position estimate, and generate dynamic capture points based on the tangential artificial potential field. S4. Dynamically group hunting agents based on auction algorithms and reinforcement learning, and complete the matching between hunting groups and hunting points; including: S41. Divide the hunting agent into... There are several encirclement groups, each corresponding to one encirclement point; for the first... The hunting agent and the first Calculate the distance between two points of encirclement; S42. Define the instantaneous value function of the hunting agent for the capture point based on the distance; define the net value function based on the global price and instantaneous value function of the capture point; determine the optimal and suboptimal capture points based on the net value function. S43. Based on the net value function, use the auction algorithm to update the local and global prices of the catch point; S5. Construct a flexible cooperative containment control law that considers actuator failures based on the matching results; including: S51, Regarding the first The first of the encirclement teams A hunting agent is defined as follows: its tracking error relative to the optimal capture point is defined as the difference between the estimated position of the hunting agent and the optimal capture point; the intra-group consistency error of the capture group is defined as the difference between the estimated positions of the hunting agents within the group; the coupling feedback function is defined as a nonlinear function describing the coupling relationship between the hunting agent and the optimal capture point. S52. Set a fault factor that considers the failure of the actuator; design an elastic cooperative containment control law based on the tracking error feedback gain, coupling feedback gain, intra-group consistency feedback gain and fault factor. S6. Based on the elastic collaborative capture control law and the state of the hunting agent, a closed-loop multi-agent elastic collaborative capture system is obtained to realize the three-dimensional spatial elastic collaborative capture of the escaped agent.

2. The three-dimensional spatial multi-agent elastic cooperative trapping control method based on reinforcement learning according to claim 1, characterized in that, S2 include: The escape agent model is rewritten to account for process noise and observation noise. The rewritten escape agent model satisfies the unknown bounded constraint; By using auxiliary variables that describe the changing trend of unknown inputs, the unknown inputs are extended into the state vector of the escape agent model to construct the enhanced state of the escape agent model. Design of continuous Kalman filters based on the escape agent model with enhanced state.

3. The three-dimensional spatial multi-agent elastic cooperative trapping control method based on reinforcement learning according to claim 1, characterized in that, S3 includes: Extract the future position estimate of the escaping agent from the future state estimate. Construct a unit virtual sphere with the future position estimate as the center, and generate a set of capture points on the unit virtual sphere. For any two capture points, construct a distance-related repulsive force, and calculate the force acting on the first capture point based on the set of projection points of the capture points on the virtual sphere. The resultant force of each capture point; the resultant force is tangentially decomposed to obtain the tangential force; the velocity and position of the capture points are updated based on the tangential force.

4. The three-dimensional spatial multi-agent elastic cooperative trapping control method based on reinforcement learning according to claim 1, characterized in that, S4 also includes constructing a comprehensive reward function for training reinforcement learning policy networks: in, To match rewards for the auction, As a reward for consistency within the group, A reward will be given for capturing the target. Penalties for unreasonable pricing To balance the grouping, and These are the weighting coefficients.

5. The three-dimensional spatial multi-agent elastic cooperative trapping control method based on reinforcement learning according to claim 1, characterized in that, In S52: The fault factors are set as follows: And satisfy: when When the actuator is working normally, it is in good working order; when At that time, the actuator partially failed; The design law for the flexible and collaborative encirclement control is as follows: in, This is an estimate of the actuator failure factor. , , These are the error feedback gain, coupling feedback gain, and intra-group consistency feedback gain, respectively; the error feedback gain, coupling feedback gain, and intra-group consistency feedback gain are updated based on the total reinforcement learning reward.

6. The three-dimensional spatial multi-agent elastic cooperative trapping control method based on reinforcement learning according to claim 1, characterized in that, S6 also includes a design group consistency feedback gain update mechanism: The effect state of the intra-group consistency feedback gain is adjusted according to the relationship between the intra-group consistency error and the preset intra-group consistency error threshold. When the intra-group consistency error does not exceed a preset threshold, the intra-group consistency feedback gain is kept updated stably according to the total reinforcement learning reward; otherwise, the intra-group consistency feedback gain is updated using the total reinforcement learning reward to enhance the intra-group consistency feedback effect.

7. The three-dimensional spatial multi-agent elastic cooperative trapping control method based on reinforcement learning according to claim 6, characterized in that, S6 also includes providing the stability conditions for the closed-loop system: Construct the following Lyapunov function: in, , , It is a symmetric positive definite matrix. and Let be the weighting coefficients, and satisfy: The first term of the Lyapunov function describes the tracking error energy of the hunting agent towards the optimal capture point, the second term describes the coupling feedback error energy, and the third term describes the intra-group consistency error energy; by appropriately selecting the control gain... , , and positive definite matrix , , Such that the derivative of the Lyapunov function satisfies: The tracking error at the capture point, the coupling feedback error, and the intragroup consistency error all gradually converge.

8. A three-dimensional spatial multi-agent elastic cooperative trapping control system based on reinforcement learning, characterized in that, include: The dynamics modeling module is used to build dynamic models of intelligent agents in three-dimensional space. The state prediction module is used to construct an enhanced state Kalman filter prediction model for an escape agent based on a three-dimensional spatial agent dynamics model. This includes extending the unknown input into the state vector of the escape agent to construct the enhanced state of the escape agent. The dynamic capture point generation module is used to extract future position estimates based on the future state estimates of the escaping agent, construct a virtual sphere centered on the future position estimates, and generate dynamic capture points based on the tangential artificial potential field. The dynamic grouping and matching module is used to dynamically group hunting agents based on auction algorithms and reinforcement learning, and to match the hunting groups with the hunting points. The cooperative control law construction module is used to construct a flexible cooperative containment control law that takes into account actuator failures based on the matching results. The closed-loop encirclement and capture execution module is used to obtain a closed-loop multi-agent elastic cooperative encirclement and capture system based on the elastic cooperative encirclement and capture control law and the state of the hunting agent, so as to realize the three-dimensional spatial elastic cooperative encirclement and capture of the escape agent. The reinforcement learning-based three-dimensional spatial multi-agent elastic cooperative trapping control system is used to implement the reinforcement learning-based three-dimensional spatial multi-agent elastic cooperative trapping control method and its steps as described in claim 1.

Images