Reinforced learning formation control method applied to swarm intelligence system
Through the distributed reinforcement learning formation control method, utilizing local information interaction and the actor-critic framework, the problem of global goal optimization in swarm intelligence systems is solved, the global performance optimization of the UAV formation is achieved, and the robustness and scalability of the system are improved.
Patent Information
- Application Number
- CN202510794043.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-13
- Publication Date
- 2025-09-23
AI Technical Summary
Existing technologies make it difficult to achieve the global optimal goal of swarm intelligence systems through local information interaction, especially in drone formation control, where there is a problem that nonlinear models are difficult to accurately obtain.
A distributed reinforcement learning formation control method is adopted. Through local information interaction between the leader and followers, and using the actor-critic reinforcement learning framework, a linear fitting of the global performance index and the update law of the optimal controller are established to achieve the optimization of the global performance index.
In the swarm intelligence system, the global performance indicators of the formation task are optimized, the communication requirements are reduced, the robustness and scalability of the system are improved, and it is suitable for collaborative tasks.
Smart Images
Figure CN120686898A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of swarm intelligence systems, and in particular to a reinforcement learning formation control method applied to swarm intelligence systems. Background Art
[0002] In recent years, formation control of swarm intelligence systems has shined in aerospace, drones, multi-robots and other fields. For example, in dangerous environments, drones are arranged into specific formations to perform reconnaissance, search, patrol and other tasks to avoid repeated reconnaissance or missed detections, thereby improving the efficiency of task execution. When drones are performing tracking and detecting target tasks, multiple drones can be arranged in a certain formation to obtain target status at different angles, thereby improving the authenticity and comprehensiveness of the information.
[0003] However, in practical applications, physical system models often contain numerous nonlinearities and cannot be accurately captured, making data-based reinforcement learning formation control methods particularly important. Furthermore, achieving optimal performance for individual agents in a swarm intelligence system is often not optimal for the overall mission objective; rather, collaboration is often required to achieve optimal global performance indicators. This can be achieved when a formation performs coordinated search and rescue, coordinated penetration, or coordinated strikes.
[0004] Therefore, it is of great practical significance to design a distributed reinforcement learning formation control method that can achieve global optimization only through local information interaction.
[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention
[0006] The present invention provides a reinforcement learning formation control method applied to swarm intelligence systems, which can realize the swarm intelligence system to form a formation while minimizing the global performance index, and can achieve the optimal pre-set system performance index while completing the task, thereby overcoming the defects existing in the existing technology to a certain extent.
[0007] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.
[0008] According to a first aspect of the present invention, a reinforcement learning formation control method for a swarm intelligence system is provided, the method comprising:
[0009] S1. Obtain the initial state information of all agents;
[0010] S2. Determine the expected target formation configuration based on the number of leaders and followers in the swarm intelligence system; the number of leaders is 1;
[0011] S3. Determine the movement trajectory of the navigator based on the initial position information of the navigator, and update the estimate of the navigator's real-time position information;
[0012] S4, determining the motion trajectory of non-neighbor followers based on the initial position information of the agent, and updating the estimate of the follower's real-time position information;
[0013] S5. Determine the distributed reinforcement learning formation control law based on the initial state information of all agents, the communication topology of the swarm intelligence system, and the real-time position information of the leader and followers;
[0014] S6. Update the real-time position information of all followers according to the reinforcement learning formation control law to complete the expected formation mission objectives.
[0015] In some exemplary embodiments, the initial state information of the agent in step S1 specifically includes:
[0016] The initial location information and initial running time of the agent, and the agent network communication topology.
[0017] In some exemplary embodiments, the expected target formation configuration determined in step S2 is specifically:
[0018] h(t)=[h1(t),h2(t),…,h N (t)] T
[0019] Among them, h(t) is the expected target formation configuration, h i (t) is the expected target formation position formed by the i-th follower, T is the matrix transpose sign, N is the number of followers, and t is the system running time.
[0020] In some exemplary embodiments, the observer for determining the motion trajectory of the navigator in step S3 is specifically:
[0021]
[0022] in, is the estimate of the leader's position information by follower i, is the estimate of the leader's position information by follower j, λ is the observer parameter, sgn is the sign function, a ij is the weight of the link composed of follower i and follower j in the communication topology, b i is the communication link between follower i and the leader, N iis the set of neighbor agents of follower i, p L For the navigator's location information, It is the estimated rate of change of real-time position information.
[0023] In some exemplary embodiments, the observer for determining the movement trajectory of the follower in step S4 is specifically:
[0024]
[0025] s ii =p i -h i
[0026] Among them, s ij is the estimate of the difference between the position information of follower i and the expected formation configuration of follower j, s kj X is the estimate of the difference between the position information of follower k and the expected formation configuration of follower j, j and μ j is a parameter greater than zero, a ik is the weight of the link composed of follower i and follower k in the communication topology, p i The real-time location information of follower i, is the estimated rate of change of the difference between the position information of follower i and the position information of follower j minus the expected formation configuration.
[0027] In some exemplary embodiments, step S5 specifically includes the following sub-steps:
[0028] S51. Based on each agent's estimate of the leader's position and the position information of other agents in the network, a linear fit of the global performance indicator and the optimal controller is established using the actor-critic reinforcement learning framework.
[0029] S52, obtaining a global performance index fitting error and an optimal controller fitting error based on the local performance index;
[0030] S53. Determine the update law of the distributed reinforcement learning formation controller based on the initial state information of all intelligent agents, the communication links of the followers in the swarm intelligence system, the real-time position information of the leader, and the processed linear fitting error.
[0031] In some exemplary embodiments, the linear fitting of the optimal controller for the global performance index in step S51 is specifically:
[0032]
[0033] z i =[z i1 ,z i2,...,z iN ]
[0034]
[0035] in, is the estimate of the global performance index by the i-th follower, k is the number of iterations, z i is the estimate of the global formation error by the ith follower, z ij is the estimate of the formation error of the j-th follower by the i-th follower, is the critic weight matrix of the i-th follower, is the actor weight matrix of the i-th follower, φ c ,φ ai is the activation function, T is the matrix transpose symbol, is the control law of the i-th follower.
[0036] In some exemplary embodiments, the linear fitting error in step S52 is specifically:
[0037]
[0038] in, is the performance index fitting error of the i-th follower, T is a small time period, Q i and R i is the local performance index parameter matrix of the i-th follower, u i is the fixed strategy of the ith follower in the T time period, m is the number of fitting error iterations, and M is the total number of fitting error iterations.
[0039] In some exemplary embodiments, the update law of the controller in step S53 is specifically:
[0040]
[0041] Where α is a constant gain, is the performance index fitting error of the i-th follower, is the optimal controller fitting error of the i-th follower, A quantity defined to achieve regularization.
[0042] According to a second aspect of the present invention, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the reinforcement learning formation control method applied to a swarm intelligence system described in the first aspect is implemented.
[0043] According to a third aspect of the present invention, a computer program product is provided, on which a computer program is stored. When the computer program is executed by a processor, the reinforcement learning formation control method applied to a swarm intelligence system described in the first aspect is implemented.
[0044] According to a fourth aspect of the present invention, there is provided an electronic device, comprising:
[0045] processor; and
[0046] a memory for storing executable instructions of the processor;
[0047] Wherein, the processor is configured to implement the reinforcement learning formation control method applied to the swarm intelligence system described in the first aspect above by executing the executable instructions.
[0048] The reinforcement learning formation control method for swarm intelligence systems provided by the embodiments of the present invention enables the swarm intelligence system to form a formation while minimizing global performance indicators, achieving optimal pre-defined system performance indicators while completing tasks. Because the present invention designs a distributed control method based on the local network weight matrix of interactions between neighboring agents, it reduces communication requirements, increases system robustness and scalability, and is more suitable for general collaborative tasks.
[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The accompanying drawings are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present invention, and together with the description, serve to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and it is clear that those skilled in the art can derive other drawings based on these drawings without inventive effort.
[0051] Figure 1 A schematic flow chart of a reinforcement learning formation control method for a swarm intelligence system provided by an embodiment of the present invention;
[0052] Figure 2 Schematic diagram of the communication topology of five drones in an embodiment of the present invention;
[0053] Figure 3 Schematic diagram of the motion trajectory of the reinforcement learning formation in an embodiment of the present invention;
[0054] Figure 4 Schematic diagram of the reinforcement learning formation error in the X direction of the follower in an embodiment of the present invention;
[0055] Figure 5 Schematic diagram of the reinforcement learning formation error in the Y direction of the follower in an embodiment of the present invention. DETAILED DESCRIPTION
[0056] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0057] This application proposes a formation control method for a swarm intelligence system, such as Figure 1 FIG. 1 is a flow chart of a formation control method for a swarm intelligence system according to an embodiment of the present application. The method includes the following steps:
[0058] Step S1: Obtain the initial state information of all agents;
[0059] In an embodiment of the present application, a swarm intelligence system established in this embodiment includes at least one leader and any number of followers. Initial state information of all agents in the swarm intelligence system is obtained, where the initial state information of the agents specifically includes the initial position information and initial running time of the agents, as well as the agent network communication topology. The agent communication topology ensures that the agents are linearly independent of each other and have certain communication capabilities, and that all agents in the target formation configuration are coplanar.
[0060] In this embodiment, the weight of the link in the communication topology G is defined as a ij , that is, the weight of the link composed of the i-th follower and the j-th follower, weight a ij Can be positive or negative. i represents the set of neighboring agents of follower i, that is, follower i and neighboring agents can communicate, which is reflected in the communication topology as being connected by communication links; in this embodiment, the communication topology G is an undirected connected graph, so there is a ij =a ji .
[0061] Step S2: determining the expected target formation configuration according to the number of followers in the swarm intelligence system;
[0062] Specifically, the expected target formation configuration is defined as:
[0063] h(t)=[h1(t),h2(t),…,h N (t)] T
[0064] Among them, h(t) is the expected target formation configuration, h i (t) is the expected target formation position formed by the i-th follower, T is the matrix transpose sign, N is the number of followers, and t is the system running time.
[0065] Step S3: determining the movement trajectory of the navigator based on the initial position information of the navigator, and updating the estimate of the navigator's real-time position information;
[0066] Specifically, the movement trajectory of the leader in the swarm intelligence system is as follows:
[0067] p L (t) = [3sin(0.4t), 2sin(0.8t)] T
[0068] Among them, p L (t) is the trajectory of the leader, 3sin(0.4t) is the horizontal coordinate trajectory of the leader, and 2sin(0.8t) is the vertical coordinate trajectory of the leader. The dynamic equation of the follower in the swarm intelligence system is:
[0069]
[0070] in, is the derivative of the follower’s real-time position information, f i (p i ), g i (p i ) is the nonlinear term of follower dynamics, u i (t) is the distributed reinforcement learning formation control law.
[0071] The motion trajectory observer of the navigator is:
[0072]
[0073] in, is the estimate of the leader's position information by follower i, λ is the observer parameter, sgn is the sign function, a ij is the weight of the link composed of follower i and follower j in the communication topology, b i is the communication link between follower i and the leader, N i is the set of neighboring agents of follower i.
[0074] Step S4: Determine the motion trajectory of the non-neighbor follower based on the initial position information of the agent, and update the estimate of the follower's real-time position information;
[0075] Specifically, the observer of the follower's motion trajectory is:
[0076]
[0077] Among them, s ij is the estimate of the difference between the position information of follower i and the expected formation configuration of follower j, s kj is the estimate of the difference between the position information of follower k and the expected formation configuration of follower j, χ j and μ j is a parameter greater than zero, s ii =p i -h i , a ik is the weight of the link consisting of follower i and follower k in the communication topology.
[0078] Step S5: Determine the distributed reinforcement learning formation control law based on the initial state information of all agents, the communication topology of the swarm intelligence system, and the real-time location information of the leader;
[0079] Step S5 specifically includes the following sub-steps:
[0080] S51. Based on each agent's estimate of the leader's position and the position information of other agents in the network, a linear fit of the global performance indicator and the optimal controller is established using the actor-critic reinforcement learning framework.
[0081] S52, obtaining a global performance index fitting error and an optimal controller fitting error based on the local performance index;
[0082] S53, based on the initial state information p of all agents i (0) Communication link a of followers in swarm intelligence system ij , real-time location information of the navigator L (t) and the linear fitting error after processing Determine the update law for a distributed reinforcement learning formation controller.
[0083] Specifically, the quadratic form of the distance between the real-time positions of all followers and the real-time position of the leader is defined as the global performance indicator of the entire drone system:
[0084]
[0085] Among them, V is the global performance index of the system, Q(z) and R are the global performance index parameter matrices, Q i and R i is the local performance index parameter matrix of the i-th follower, J i (z,u i) is the local performance indicator of the i-th follower. Traditional algorithms require that each agent’s local optimum is the global optimum, but considering the above cooperative problem, a distributed algorithm is designed.
[0086] The Hamiltonian function of this performance indicator is:
[0087]
[0088] in, is the partial derivative of the system’s global performance index with respect to the formation error of the i-th follower, is the derivative of the navigator's dynamics.
[0089] By taking the partial derivative of the Hamiltonian function, we can get the optimal controller:
[0090]
[0091] in, is the inverse matrix of the performance index parameter matrix of the i-th follower;
[0092] Each follower uses its own estimate s of other followers ij and estimates of the navigator To get an estimate of the global formation error:
[0093] z i =[z i1 ,z i2 ,...,z iN ]
[0094]
[0095] Furthermore, each leader uses its own parameter network weight W c,i and local performance index J i (z,u i ), use the actor-critic reinforcement learning framework to obtain a local estimate of the global performance indicator function, and establish a linear fit of the global performance indicator and the optimal controller:
[0096]
[0097] in, is the estimate of the global performance index by the i-th follower, k is the number of iterations, z i is the estimate of the global formation error by the ith follower, is the critic weight matrix of the i-th follower, is the actor weight matrix of the i-th follower, φ c ,φ ai is the activation function, and T is the matrix transpose symbol.
[0098] Furthermore, the linear fitting error of the global performance index and the linear fitting error of the optimal controller are obtained:
[0099]
[0100] in, is the performance index fitting error of the i-th follower, T is a small time period, Q i and R i is the local performance index parameter matrix of the i-th follower, u i is the fixed strategy of the ith follower in the T time period, m is the number of fitting error iterations, and M is the total number of fitting error iterations.
[0101] The performance index fitting error is It can be obtained based on local information, but the optimal controller fitting error requires the mean of all followers, which can be obtained through M iterations.
[0102] Furthermore, based on the regularized gradient descent method, the update law of the distributed reinforcement learning formation controller is determined:
[0103]
[0104]
[0105] Where α is a constant gain, is the performance index fitting error of the i-th follower, is the optimal controller fitting error of the i-th follower, A quantity defined to achieve regularization.
[0106] S6. Update the real-time position information of all followers according to the reinforcement learning formation control law to complete the expected formation mission objectives.
[0107] In order to better understand the technical method of this application, the following specific embodiments are proposed in this application:
[0108] Assume five drones are distributed across a plane, with one leader and four followers. The formation task is as follows: the leader has its own dynamic model and flies according to a given trajectory. To ensure the followers can track the leader's position as quickly as possible, the quadratic form of the distance between the followers' real-time positions and the leader's is used as the performance indicator for the entire drone system. This means that while completing the formation task, the followers must minimize travel distance and fuel consumption, ensuring high practical application value.
[0109] In a two-dimensional plane, given the position information of the leader M = 1 and follower N = 4 drones:
[0110] p L (0)=[0,0] T m,p1(0)=[5,4] T m,p2(0)=[-5,5] T m,
[0111] p3(0)=[4,-3] T m,p4(0)=[-5,-5] T m,
[0112] The motion trajectory of the dynamic navigator agent is:
[0113] p L (t) = [3sin(0.4t), 2sin(0.8t)] T
[0114] The dynamic equation of the follower agent is:
[0115]
[0116] Among them, α i＝1,2,3,4 =-0.5,0.7,-0.2,0.1,β i＝1,2,3,4 =-2,0.3,5,-8,p ix is the x-axis coordinate, p iy is the coordinate of the y-axis;
[0117] The expected formation configuration is:
[0118] h1=[4;4], h2=[-4;4], h3=[4;-4], h4=[-4;-4],
[0119] Initialize the UAV simulation running time t=0s and set the UAV formation simulation end time t=20s.
[0120] Among the five preset drones, one is selected as the leader, denoted as No. 0. The remaining four drones are used as followers in the drone intelligent system reinforcement learning formation, denoted as No. 1, No. 2, No. 3, and No. 4. Figure 2 The undirected communication topology graph G of the four drones shown is used to determine the drone cluster system formation (G, h) based on the task requirements of the drone intelligent system reinforcement learning formation.
[0121] The trajectory planning of the leader of the UAV intelligent system reinforcement learning formation can be planned in advance. In this embodiment, the leader's desired trajectory is a circular trajectory. By giving the initial time t = 0 and performing numerical simulation using the numerical simulation software MATLAB, a series of discrete points are obtained and the trajectory diagram is drawn using MATLAB.
[0122] The quadratic form based on the distance between the real-time position of the follower and the real-time position of the leader is used as the performance indicator of the entire drone system:
[0123]
[0124] Among them, R1=R2=R3=R4=I2, and I2 is a two-dimensional unit matrix.
[0125] According to the information interaction between neighbors, the motion trajectory observer of the navigator is obtained as:
[0126]
[0127] According to the information interaction between neighbors, the observer of the follower's trajectory is obtained as:
[0128]
[0129] According to the observer information, take offline strategy u i Run for a time interval of T to obtain the linear fitting error of the global performance index and the linear fitting error of the optimal controller:
[0130]
[0131] Based on the regularized gradient descent and reinforcement learning interaction weight matrix method, the update of the reinforcement learning formation control law of the distributed UAV intelligent system is obtained as follows:
[0132]
[0133] Based on the above observer, controller, update law, and the dynamic equation of the follower agent, the expected UAV intelligent system formation can be achieved.
[0134] This embodiment simulates the UAV intelligent system under the reinforcement learning formation control law, such as Figures 3 to 5 As shown in the figure, X is the horizontal coordinate in two-dimensional space, and Y is the vertical coordinate in two-dimensional space. h is the expected formation configuration of the UAV intelligent system followers. t is the simulation run time of the UAV intelligent system formation. x is the X position of the UAV in the UAV intelligent system formation. y is the Y position of the UAV in the UAV intelligent system formation.
[0135] Figure 3This figure shows the real-time motion trajectory of a UAV intelligent system using reinforcement learning in a two-dimensional space. The leader is pre-programmed to perform an elliptical motion around the origin of the coordinate system. Initially, at time t = 0s, the five UAVs are located at different locations in the same space. The followers, using the formation control law from step 5, track the leader. The four UAVs accurately form the pre-programmed diamond formation. Figure 3 The three moments t=0s, t=10s and t=18s were selected and the positions of the leader and follower were marked, showing that under the action of the reinforcement learning controller, the UAV intelligent system achieved the desired formation configuration. Figure 4 The distance between the horizontal axis position of the follower and the expected trajectory in the UAV intelligent system is shown. Figure 5 The distance between the longitudinal axis position of the follower in the UAV intelligent system and the desired trajectory is shown. It is clearly shown that as the formation is completed, the formation error converges to a bounded error, demonstrating that the reinforcement learning formation control law designed in this invention can effectively achieve the target task in the presence of nonlinearities. These results demonstrate that the distributed reinforcement learning formation control law designed in this invention for the UAV intelligent system can accurately complete tasks such as formation and formation closing, based on actual conditions. It also ensures the optimality of additional performance indicators while completing the formation task, resisting the influence of system nonlinearities.
[0136] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.
[0137] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the claims.
[0138] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings and that various modifications and variations can be made without departing from the scope thereof, which is limited only by the appended claims.
Claims
1. A reinforcement learning formation control method applied to a swarm intelligence system, characterized in that: The method comprises: S1. Obtain the initial state information of all agents; S2. Determine the expected target formation configuration based on the number of leaders and followers in the swarm intelligence system; the number of leaders is 1; S3. Determine the movement trajectory of the navigator based on the initial position information of the navigator, and update the estimate of the navigator's real-time position information; S4, determining the motion trajectory of non-neighbor followers based on the initial position information of the agent, and updating the estimate of the follower's real-time position information; S5. Determine the distributed reinforcement learning formation control law based on the initial state information of all agents, the communication topology of the swarm intelligence system, and the real-time position information of the leader and followers; S6. Update the real-time position information of all followers according to the reinforcement learning formation control law to complete the expected formation mission objectives.
2. The reinforcement learning formation control method applied to a swarm intelligence system according to claim 1, characterized in that: The initial state information of the agent in step S1 specifically includes: The initial location information and initial running time of the agent, and the agent network communication topology.
3. The reinforcement learning formation control method for a swarm intelligence system according to claim 1, characterized in that: The expected target formation configuration determined in step S2 is specifically: h(t)=[h1(t),h2(t),…,h N (t)] T Among them, h(t) is the expected target formation configuration, h i (t) is the expected target formation position formed by the i-th follower, T is the matrix transpose sign, N is the number of followers, and t is the system running time.
4. The reinforcement learning formation control method for a swarm intelligence system according to claim 1, characterized in that: The observer for determining the movement trajectory of the navigator in step S3 is specifically: in, is the estimate of the leader's position information by follower i, is the estimate of the leader's position information by follower j, λ is the observer parameter, sgn is the sign function, a ij is the weight of the link composed of follower i and follower j in the communication topology, b i is the communication link between follower i and the leader, N i is the set of neighbor agents of follower i, p L For the navigator's location information, It is the estimated rate of change of real-time position information.
5. The reinforcement learning formation control method for a swarm intelligence system according to claim 1, characterized in that: The observer for determining the follower's motion trajectory in step S4 is specifically: s ii =p i -h i Among them, s ij is the estimate of the difference between the position information of follower i and the expected formation configuration of follower j, s kj is the estimate of the difference between the position information of follower k and the expected formation configuration of follower j, χ j and μ j is a parameter greater than zero, a ik is the weight of the link composed of follower i and follower k in the communication topology, p i The real-time location information of follower i, is the estimated rate of change of the difference between the position information of follower i and the position information of follower j minus the expected formation configuration.
6. The reinforcement learning formation control method for a swarm intelligence system according to claim 1, characterized in that: The step S5 specifically includes the following sub-steps: S51. Based on each agent's estimate of the leader's position and the position information of other agents in the network, a linear fit of the global performance indicator and the optimal controller is established using the actor-critic reinforcement learning framework. S52, obtaining a global performance index fitting error and an optimal controller fitting error based on the local performance index; S53. Determine the update law of the distributed reinforcement learning formation controller based on the initial state information of all intelligent agents, the communication links of the followers in the swarm intelligence system, the real-time position information of the leader, and the processed linear fitting error.
7. The reinforcement learning formation control method for a swarm intelligence system according to claim 6, characterized in that: In step S51, the linear fitting of the optimal controller for the global performance index is specifically as follows: With i =[z i1 ,With i2 ,...,With iN ] in, is the estimate of the global performance index by the i-th follower, k is the number of iterations, z i is the estimate of the global formation error by the ith follower, z ij is the estimate of the formation error of the j-th follower by the i-th follower, is the critic weight matrix of the i-th follower, is the actor weight matrix of the i-th follower, φ c ,φ ai is the activation function, T is the matrix transpose symbol, is the control law of the i-th follower.
8. The reinforcement learning formation control method for a swarm intelligence system according to claim 7, characterized in that: The linear fitting error in step S52 is specifically: in, is the performance index fitting error of the i-th follower, T is a small time period, Q i and R i is the local performance index parameter matrix of the i-th follower, u i is the fixed strategy of the ith follower in the T time period, m is the number of fitting error iterations, and M is the total number of fitting error iterations.
9. The reinforcement learning formation control method for a swarm intelligence system according to claim 8, characterized in that: The update law of the controller in step S53 is specifically: Where α is a constant gain, is the performance index fitting error of the i-th follower, is the optimal controller fitting error of the i-th follower, A quantity defined to achieve regularization.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the phased array radar design method according to any one of claims 1 to 9 is implemented.