Formation control method for air-ground cluster system for cooperative search of complex environment target

Abstract

The application discloses a kind of air-ground cluster system formation control methods for complex environment target cooperative search. The method is for the air-ground heterogeneous unmanned cluster composed of leader unmanned aerial vehicle and follower unmanned vehicle, air formation control and ground encirclement control are set respectively, and double-strategy iterative reinforcement learning is used to optimize control strategy. In the optimization process, an additional stability term is introduced to construct a data matrix, which adjusts the data matrix as the double-strategy switching boundary parameter changes;Generalized Hilbert-vester transpose matrix equation based on data is constructed and positive definite solution matrix is solved to determine the minimum switching boundary parameter. Based on the parameter, stable strategy iteration and optimal strategy iteration are executed in stages to obtain the optimal control strategy acting on the cluster. The application can realize strategy learning initialization and control performance improvement without pre-set model-based initial stable strategy.

Description

Technical Field

[0001] This invention relates to the fields of reinforcement learning and unmanned swarm systems, specifically to a formation control method for air-to-ground swarm systems for collaborative target search in complex environments. Background Technology

[0002] With the rapid development of multi-agent systems and unmanned equipment technologies, air-to-ground heterogeneous unmanned swarm systems composed of UAVs and unmanned vehicles have broad application prospects in complex mission scenarios such as surveillance and reconnaissance, target search, smart city inspection, and disaster relief. Compared with single-type unmanned systems, air-to-ground heterogeneous unmanned swarm systems can comprehensively leverage the advantages of UAVs (wide field of view and high mobility) and unmanned vehicles (high carrying capacity and good ground continuous operation capability), thereby improving the efficiency of target collaborative search and mission execution in complex environments.

[0003] In heterogeneous air-to-ground unmanned swarm systems, formation encirclement control is a crucial technology for achieving collaborative swarm operations. Its purpose is to enable the leader drone to form a target formation that meets mission requirements, while ensuring that the states or outputs of follower drones converge within the convex hull formed by the leader drone's states, under conditions where each agent can only acquire local neighborhood information. This achieves unified control of aerial formation guidance and ground encirclement collaboration. Through this control method, the air-to-ground swarm system can maintain a predetermined spatial structure and collaborative relationships in applications such as target search, area encirclement, path coordination, and dynamic task allocation.

[0004] However, existing formation encirclement control methods typically rely on relatively accurate system dynamics models and are mostly designed for single-domain isomorphic or weakly heterogeneous multi-agent systems. In complex environment target cooperative search scenarios, air-to-ground unmanned systems are characterized by large differences in dynamic models, changes in communication topology, uncertainty of external disturbances, and dynamic changes in the mission environment, making high-precision modeling difficult to achieve. Traditional model-driven control methods are prone to insufficient adaptability, decreased robustness, and difficulty in optimizing control performance under these conditions. Furthermore, existing methods often focus more on the steady-state convergence performance of the system, while neglecting the overall trajectory control of the swarm, transient maneuver response, and cooperative search efficiency in complex environments, thus limiting their application in practical air-to-ground cooperative missions.

[0005] To reduce reliance on precise models, data-driven control methods such as reinforcement learning and adaptive dynamic programming are increasingly being introduced into the field of unmanned swarm cooperative control. These methods can iteratively learn control strategies using system operational data, obtaining near-optimal control strategies even under unknown or incompletely known system model conditions, thus providing a new technical approach for air-ground cooperative control in complex environments.

[0006] However, existing reinforcement learning-based cooperative control methods still have certain shortcomings. On the one hand, some methods require an initial stable control policy to be given in advance during policy iteration or value iteration. The construction of the initial stable policy usually still relies on system model information or human experience, which is difficult to meet the practical application requirements under unknown model conditions. On the other hand, some methods have high requirements for the full-rank condition of the collected data and require the construction of high-dimensional matrices or complex matrix operations. This makes the computational and storage requirements increase significantly with the increase of system scale, reducing their real-time performance and scalability in large-scale air-ground heterogeneous unmanned swarm systems.

[0007] Therefore, there is an urgent need to propose a formation control method for air-to-ground swarm systems that is oriented towards collaborative target search in complex environments. This method would allow the optimal control gain matrix to be learned through a data-driven approach without prior initial stability control strategies, even when the system dynamics model is unknown. This method would also improve the collaborative search capability, maneuverability, and computational efficiency of air-to-ground swarm systems in complex environments while satisfying formation encirclement control constraints. Summary of the Invention

[0008] In view of this, the present invention provides a formation control method for air-to-ground swarm systems for collaborative target search in complex environments, which solves the problems of existing air-to-ground heterogeneous unmanned swarms relying on initial stability strategies, having high computational complexity, and being unable to balance stability and optimality in formation encirclement control under conditions where the dynamic model is unknown or difficult to model accurately.

[0009] The method described in this invention is applicable to air-to-ground heterogeneous unmanned swarm systems consisting of a leader UAV and follower UAVs. This method constructs a two-layer distributed observer to achieve distributed estimation of the formation state of the leader UAV and the encirclement state of the follower UAVs, respectively. Based on this, a data-driven offline policy reinforcement learning mechanism is used to iteratively optimize the control strategies at the formation and encirclement layers to obtain the corresponding optimal control gain matrices, thereby achieving coordinated control of aerial UAV formation guidance and ground-based UAV encirclement tracking.

[0010] Furthermore, this invention introduces additional stabilizing terms and adaptive parameters during the strategy iteration process, enabling the control strategy to meet the closed-loop stability requirements without needing to pre-define an initial stable control strategy. By setting a dynamically adjustable switching boundary, the strategy learning process is divided into a stable iteration stage and an optimal iteration stage, allowing the control strategy to first meet the stability requirements and then gradually converge towards the optimal control strategy.

[0011] Furthermore, this invention employs the generalized Hillwest transpose matrix equation to solve for the control gain matrix, avoiding the high-dimensional matrix construction and large-scale matrix operations caused by the Kronecker product in traditional reinforcement learning methods. This reduces data storage and computational overhead, and improves the algorithm's operating efficiency and scalability in air-ground heterogeneous unmanned swarm systems.

[0012] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0013] A formation control method for an air-to-ground swarm system for collaborative target search in complex environments is disclosed. In the air-to-ground swarm system, a virtual leader issues a global search command, and the leader UAV forms an aerial observation array according to the desired formation to conduct aerial searches in complex environments. When a suspected target is detected, the aerial observation array moves as a whole, driving the convex hull area to synchronously move towards the target area. Follower UAVs maintain communication and linkage with the leader UAV, and their movement range is always constrained within the convex hull area. They move with the convex hull to approach the target, realizing the formation encirclement and collaborative control of the air-to-ground swarm. This method employs a dual-strategy iterative reinforcement learning approach to optimize the control strategy, including: System operation data under segmented constant continuous excitation is collected, and the system operation data is combined with additional stabilization terms to construct a data matrix. The additional stabilization terms have a corresponding conversion relationship with the dual-strategy switching boundary parameters, so that the data matrix changes with the switching boundary parameters, thereby realizing strategy learning initialization without a preset model-based initial stabilization strategy. A generalized Hill West transpose matrix equation based on data is constructed and substituted into the data matrix for solving. The minimum switching boundary parameter that makes the matrix equation produce a positive definite solution matrix is ​​determined and used as the dual-strategy switching boundary. Based on the dual-strategy switching boundary, stable strategy iteration and optimal strategy iteration are performed in stages, and the learned optimal control strategy is applied to the air-to-ground cluster system.

[0014] Preferably, the specific process of constructing and iteratively optimizing the formation control strategy of the air-to-ground cluster system is as follows: Step 1: Establish dynamic models for the virtual leader, leader drone, and follower drone, and combine them with the desired formation to construct a globally unified description model of the air-to-ground cluster system. Step 2: Construct a two-layer distributed observer, in which the formation layer observer is used to estimate the formation reference trajectory information generated by the virtual leader and the desired formation, and the encirclement layer observer is used to estimate the encirclement reference state information located within the convex hull formed by the leader; Step 3: Combining the global unified description model of the cluster system and the two-layer distributed observer, construct the formation layer augmentation system and the encirclement layer augmentation system respectively, and design the corresponding distributed state feedback controller accordingly; Step 4: For the formation layer controller and the encirclement layer controller, a dual-strategy iterative reinforcement learning algorithm is used to learn and optimize the control strategy, specifically including: Step 41: Apply piecewise constant and continuous excitation input to the air-to-ground cluster system, collect system operation data and perform integration processing to construct a first data storage matrix without Kronecker product form; Step 42: Introduce additional stabilization terms to preprocess the collected system operation data to construct a second data storage matrix for the state differential of the air-to-ground cluster system; select a set of linearly independent column vector data from the first data storage matrix and the second data matrix according to the data index to construct a small-volume non-singular data matrix. Step 43: Construct the generalized Hillwest transpose matrix equation based on the non-singular data matrix and solve the Riccati equation solution matrix; gradually increase the switching boundary parameter according to a preset monotonically increasing sequence, wherein the changed switching boundary parameter will affect the value of the additional stabilizing term, thereby changing the data in the non-singular data matrix; reconstruct the non-singular data matrix based on the updated additional stabilizing term, and repeatedly solve the generalized Hillwest transpose matrix equation using the new non-singular data matrix until the switching boundary parameter that makes the Riccati solution matrix satisfy the positive definite condition is obtained, and it is determined as the dual-policy switching boundary, so that the non-singular data matrix constructed under the corresponding additional stabilizing term satisfies the stability policy learning initialization requirements of the reinforcement learning algorithm; Step 44: First-layer policy iteration: Learning a stable control gain matrix; When the adaptive parameters do not reach the switching boundary parameters, the first-layer strategy iterative learning is performed to learn the stable control gain matrix: the current additional stable term is calculated based on the switching boundary parameters and the adaptive parameters of the current iteration round; a non-singular data matrix is ​​constructed based on the current additional stable term; and the control gain matrix and the solution matrix of the Riccati equation are solved by substituting the generalized Hillwest transpose matrix equation to learn the stable control gain matrix; and the adaptive parameters of the next iteration round are updated and calculated using the solved control gain matrix and the solution matrix of the Riccati equation. Step 45: Second-layer policy iteration: Learning the optimal control gain matrix; When the adaptive parameter reaches the switching boundary parameter, switch to the second-level strategy to iteratively learn the optimal control gain matrix: set the current additional stabilizing term to zero, construct a non-singular data matrix based on the zeroed additional stabilizing term, and substitute it into the generalized Hillwest transpose matrix equation and solve iteratively to learn the optimal control gain matrix. Step 5: Based on the optimal control gain matrix obtained from the final learning, obtain the optimal control strategy and apply the optimal control strategy to the air-ground swarm system to achieve optimal formation and encirclement cooperative control of the air-ground heterogeneous unmanned swarm of UAVs and unmanned vehicles, and complete the cooperative target search in complex environments.

[0015] Preferably, in step 42, an additional stabilization term is introduced to preprocess the collected system operation data and construct a second data storage matrix for the state differential of the air-to-ground cluster system, specifically including: Superscript of the defined parameter Indicates the first The next iteration; the subscript of the parameter In the air-to-ground cluster system, the first i The leader drone's first An augmentation subsystem, subscript Indicates the first The first follower driverless car One augmentation subsystem; and These are the corresponding augmentation subsystems. and Additional stabilizing terms; The data sampling time series for the leader drone and the follower drone are set as follows: and ; Using additional stabilizing terms and The collected system operation data is preprocessed by integration to construct a second data storage matrix for the state differential of the air-to-ground cluster system. and ,in: in, The data stored contains the differential state of the leader drone during the operation of the air-to-ground swarm augmentation system. Related system operation data, It is the first i The leader drone's first Augmented vectors of augmented subsystems yes The differential, The parentheses after the parenthesis Indicates the sampling time. The subscript indicates the sequence number in the sampling time series; The data stored contains the state differentials between the air-to-ground cluster augmentation system and the follower unmanned vehicle during operation. Related system operation data; It is the first The first follower driverless car Augmented vectors of augmented subsystems yes The differential; This is the integration step size.

[0016] Preferably, step 43 specifically includes: The switching boundary parameters are set to the initial values ​​of a preset monotonically increasing sequence, and the initial control gain matrix is ​​set to zero. Based on the corresponding conversion relationship between the additional stable terms and the switching boundary parameters, the current additional stable terms of the leader UAV augmentation subsystem and the follower UAV augmentation subsystem are determined respectively. Based on the current additional stable terms, the second data storage matrix is ​​constructed using the construction method in step 42, and combined with the first data storage matrix, the current non-singular data matrix is ​​obtained. Substitute the current non-singular data matrix into the generalized Hillwest transpose matrix equation to solve the matrix equation solution matrix containing the Riccati equation solution matrix and the control gain matrix; The positive definiteness of the solution matrix of the Riccati equation is then verified. If the solution matrix is ​​positive definite, the current switching boundary parameters ensure that the system satisfies the Hurwitz stability condition, and the current switching boundary parameters are determined as the dual-strategy switching boundary. If the solution matrix is ​​not positive definite, the value of the switching boundary parameters is gradually increased according to a preset monotonically increasing sequence, and the current additional stable term and the current non-singular data matrix are recalculated. The matrix equation is then repeatedly solved until the switching boundary parameters that make the solution matrix of the Riccati equation positive definite are obtained. , to use it as the boundary for switching between the two strategies; The conversion relationship between the additional stability term and the switching boundary parameter is as follows: ;

[0017] Among them, superscript Indicates the first Next iteration; subscript In the air-to-ground cluster system, the first i The leader drone's first An augmentation subsystem; subscript Indicates the first The first follower driverless car One augmentation subsystem; and These are the corresponding augmentation subsystems. and Additional stabilizing terms; and They represent the corresponding augmented subsystems. and Switching boundary parameters; and They represent the corresponding augmented subsystems. and Adaptive parameters can be learned and optimized during the iteration process; and Represents the identity matrix, subscript It is the identity matrix The order of represents the corresponding augmented subsystem. The state dimension; It is the identity matrix The order of represents the corresponding augmented subsystem. The state dimension; Wherein, the adaptive parameter and The update law is:

[0018] in, and To update the step size iteratively; and These are the first air-to-ground cluster systems. The leader drone's first During the iterative learning process of the augmented subsystem, the first... The control gain matrix of the second iteration and the first iteration The solution matrix of the Riccati equation in the next iteration; and The first The first follower driverless car During the iterative learning process of the augmented subsystem, the first... The control gain matrix of the second iteration and the first iteration The solution matrix of the Riccati equation in the next iteration. and Let represent the minimum and maximum eigenvalues ​​of the matrix, respectively. and For auxiliary weight matrix; and The first The leader drone's first The augmentation subsystem and the first The first follower driverless car The state weight matrix corresponding to each augmented subsystem and The first The leader drone's first The augmentation subsystem and the first The first follower driverless car The control weight matrix corresponding to each augmented subsystem; For augmentation subsystem The corresponding output weight matrix, For augmentation subsystem The corresponding output matrix; For augmentation subsystem The corresponding output weight matrix, For augmentation subsystem The corresponding output matrix; and This is an intermediate quantity.

[0019] Preferably, step one specifically includes: (1) Constructing the first air-to-ground cluster system i Dynamics model of a leader drone:

[0020] in, For leaders to assemble drones, This represents the total number of drones belonging to the leader, and is also the largest drone number. and The first i The state vector and control inputs of the leader drone; and These are the corresponding system matrix and input matrix, respectively; for The derivative; (2) Constructing the first air-to-ground cluster system Dynamics model of a follower autonomous vehicle:

[0021] in, To gather the following driverless cars, The total number of follower drones, followed by the leader drone's number; and The first The state vector and control input of a follower autonomous vehicle; and These are the corresponding system matrix and input matrix, respectively; (3) Construct a dynamic model of the virtual leader in the air-to-ground cluster system:

[0022] in, and These represent the system state and bounded control input of the virtual leader, respectively. and These are the system matrix and input matrix of the virtual leader, respectively; (4) Based on the task requirements, construct the desired formation in the air-to-ground cluster system:

[0023] in, For the first i The and the first j The expected state deviation between the leader drones; and The first i The and the first j The expected state deviation between the leader drone and the virtual leader; (5) Based on the dynamic models of the leader drone, follower drones, and virtual leader, the desired formation is combined to construct a globally unified description model, including the following subsystem forms:

[0024] in, and These represent the virtual leaders. The system state, control inputs, and system outputs of each subsystem. , and These represent the virtual leaders. The system matrix, input matrix, and output matrix of each subsystem, with subscripts. ,in These represent the three-dimensional position and yaw angle dimensions, respectively. and In the air-to-ground cluster system, the first The leader drone The status, control inputs, and outputs of each subsystem and These are the dynamic matrices that match the leader drone dynamics model; and In the air-to-ground cluster system, the first The first follower driverless car The status, control inputs, and outputs of each subsystem and These are the dynamic matrices that match the dynamics model of the follower autonomous vehicle; For the first and Among the leaders of the drones, the first Dimensional state deviation, and The first i and jThe leader drone and the virtual leader are the first Dimensional state deviation; , These represent the three-dimensional position and yaw angle, respectively.

[0025] Preferably, in step two, a two-layer distributed observer is constructed based on the symbolic function, the two-layer distributed observer including a formation layer observer and an encirclement layer observer; The formation layer observer is used to estimate the state of the leader UAV in an air-to-ground heterogeneous unmanned swarm system. The formation layer observer is represented as follows:

[0026] in, and These represent the first [unit / item] used in the air-to-ground cluster system. Each leader drone provides state and control inputs to a formation layer observer with distributed state estimation; For formation layer observation status The derivative; and It is a positive gain constant. It is a symbolic function; For the control gain of the formation layer observer, and By positive definite symmetric matrix Determined, wherein the positive definite symmetric matrix By solving linear matrix inequality equations get; , It is the information transmission matrix corresponding to the communication network topology diagram between the virtual leader and the leader drone in an air-to-ground unmanned swarm system. Represents the smallest eigenvalue of the matrix; Indicates the first i Neighborhood formation following error of a leader drone; For Leaders Drones A collection of neighboring intelligent agents; For the first The leader drone and the first The adjacency weight of the communication topology between the leader drones, if , When a communication connection exists, Greater than zero, otherwise It equals zero; For virtual leadership and leader drones The connection weight between them; if a communication connection exists between them, then... Greater than zero, otherwise It equals zero; The encirclement observer is used to perform distributed estimation of the state of follower unmanned vehicles in an air-to-ground heterogeneous unmanned swarm system. The encirclement layer observer is represented as follows:

[0027] in, and These represent the first [unit / item] used in the air-to-ground unmanned swarm system. Each follower autonomous vehicle provides state and control inputs to an enclosing layer observer for distributed state estimation. Enclosure layer observer status The derivative; For the first The neighborhood encirclement and following error of a follower autonomous vehicle; For follower driverless cars A collection of neighboring intelligent agents; and It is a positive gain constant; It is a symbolic function; For the control gain of the enclosing layer observer, and By positive definite symmetric matrix Determined, wherein the symmetric matrix By solving linear matrix inequality equations get, , It is the information transmission matrix corresponding to the communication network topology between the leader drone and the follower drones in an air-to-ground unmanned swarm system. For the first and The adjacency weight of the communication topology between each follower autonomous vehicle, if and If a communication connection exists, then Greater than zero, otherwise It equals zero; To lead drones With follower driverless cars The connection weight between them; if a communication connection exists between them, then... Greater than zero, otherwise It equals zero.

[0028] Preferably, in step four, the method for determining the completion of the control strategy iterative optimization includes: If the difference between the control gain matrix obtained in the current iteration and the control gain matrix obtained in the previous iteration is less than a preset convergence threshold, then the control strategy iteration optimization is completed and the iteration is stopped. The optimal control gain matrix corresponding to the leader UAV and the follower UAV in the air-ground heterogeneous unmanned swarm system is obtained, and then the optimal formation controller and the optimal encirclement controller at the formation layer and encirclement layer are obtained.

[0029] The present invention also provides an electronic device, comprising: a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and when the program or instructions are executed by the processor, the steps of the above-described air-to-ground cluster system formation control method for cooperative target search in complex environments are implemented.

[0030] The present invention further provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the above-described air-to-ground cluster system formation control method for cooperative target search in complex environments.

[0031] Beneficial effects: (1) This invention is aimed at the cross-domain collaborative scenario of air and ground integration. Without relying on the precise dynamic model, it uses system operation data to learn the control strategies of the formation layer and the encirclement layer, so that the air-ground heterogeneous unmanned swarm can realize the optimal formation and encirclement control driven by data, thereby improving the collaborative efficiency and maneuver response capability in the collaborative search process of targets in complex environments.

[0032] (2) In the reinforcement learning strategy iteration process, the present invention introduces additional stabilizing terms and adaptive parameters. Without the need for manual preset of the initial stabilizing control strategy, the poles of the closed-loop system can be adjusted by the additional stabilizing terms, so that the closed-loop system in the early stage of iteration meets the stability requirements, thereby reducing the algorithm's dependence on prior system models and human experience.

[0033] (3) By setting a dynamically adjustable switching boundary, the present invention divides the control strategy iteration process into a stable iteration stage and an optimal iteration stage, so that the algorithm first completes the learning of the stable control strategy and then smoothly switches to the optimal control strategy iteration, thereby taking into account both the system operation stability and the optimal control performance, and realizing the coordinated optimization of the formation control target and the encirclement control target.

[0034] (4) The present invention solves the control gain matrix based on the generalized Hillwest transpose matrix equation, avoiding the use of Kronecker product to construct a high-dimensional data matrix. This can reduce the data storage and computational overhead caused by the expansion of matrix dimension, improve the efficiency of iterative solution of control strategy, and is more suitable for large-scale air-ground heterogeneous unmanned cluster system composed of multiple UAVs and multiple unmanned vehicles.

[0035] (5) The present invention is adapted to the collaborative operation architecture of air-ground heterogeneous unmanned clusters. Through a two-layer distributed observer, a unified dynamic model and a hierarchical controller, a complete control link is formed, which can support the operation mode of combining UAV aerial formation search and unmanned vehicle ground close encirclement tracking, thereby improving the execution stability and applicability of target collaborative search tasks in complex environments such as complex mountains and disaster areas. Attached Figure Description

[0036] Figure 1 A flowchart illustrating a formation control method for an air-to-ground cluster system in a complex environment for collaborative target search, provided as an embodiment of this application; Figure 2 A schematic diagram illustrating the formation and encirclement control process of an air-to-ground cluster system provided in this application embodiment; Figure 3 This is a schematic diagram of the hardware structure of a computer device for executing the air-to-ground cluster system formation control method, provided as an embodiment of this application. Detailed Implementation

[0037] The air-to-ground heterogeneous unmanned swarm system provided in this embodiment of the invention includes one virtual leader. L A leader drone and F There are several follower unmanned vehicles. The virtual leader, leader drones, and follower unmanned vehicles interact with each other through a communication network topology. Specifically, the communication network topology between the virtual leader and leader drones is an undirected graph, and there are directed paths from the virtual leader to each leader drone; the communication network topology between the leader drones and follower unmanned vehicles is also an undirected graph, and there are directed paths from each leader drone to each follower unmanned vehicle.

[0038] For collaborative target search tasks in complex environments, the air-to-ground heterogeneous unmanned swarm system adopts a hierarchical collaborative structure. A virtual leader receives and carries global search instructions from the mission planning layer. These instructions include key information such as search area boundaries, reference tracks, and target guidance directions, and are transmitted to each leader UAV via a communication network. L Each leader drone, leveraging its wide aerial observation range and high maneuverability, forms an aerial observation array according to a desired formation, conducting extensive coarse searches of complex terrain. By maintaining a pre-defined formation configuration, the leader drone swarm can reduce blind spots and overlapping search areas, improving search coverage efficiency. For example, Figure 2 As shown, the four leader drones, initially dispersed, form and maintain a rectangular formation after being controlled by formation to perform a coarse search mission. FUnder communication topology constraints and encirclement control, the motion state of a follower drone is restricted to the convex hull formed by the state of the leader drone. For example, Figure 2 As shown, three follower UAVs enter and remain within the convex hull area based on their position. This encirclement control method allows the follower UAVs to remain within areas already covered by aerial UAV observations, reducing the risk of entering unexplored areas under limited visibility conditions. It also maintains communication and information guidance with the leader UAV, achieving coordinated integration of aerial coarse search and ground fine search at both spatial and informational levels. When the aerial observation array of the leader UAVs detects a suspected target in a certain area, the convex hull area defined by the leader UAVs' states dynamically adjusts and moves towards the vicinity of the suspected target as the leader UAV formation moves, guiding the follower UAVs within the convex hull to converge towards the suspected target's location. Leveraging their close proximity to the ground and ease of close-range observation, the follower UAVs perform close-range observation, identification, and continuous tracking of the suspected target.

[0039] To achieve the above-mentioned air-ground cooperative control process, this invention provides a formation control method for air-ground cluster systems oriented towards cooperative target search in complex environments. This method is based on a dual-strategy iterative reinforcement learning algorithm and specifically includes: collecting operational data of an air-to-ground heterogeneous unmanned swarm system under piecewise constant continuous excitation; constructing a data matrix based on the system operational data and an introduced additional stabilization term, wherein the additional stabilization term and the dual-strategy switching boundary parameters have a corresponding conversion relationship, so that the data matrix changes with the dual-strategy switching boundary parameters to achieve strategy learning initialization without a preset model-based initial stabilization strategy; constructing a data-based generalized Hillwest transpose matrix equation and substituting the data matrix constructed based on the additional stabilization term into the generalized Hillwest transpose matrix equation to obtain the corresponding solution matrix; determining the minimum switching boundary parameter that makes the generalized Hillwest transpose matrix equation have a positive definite solution matrix by changing the dual-strategy switching boundary parameter, and using this minimum switching boundary parameter as the dual-strategy switching boundary; performing stable strategy iteration and optimal strategy iteration in stages based on the dual-strategy switching boundary; applying the optimal control strategy obtained by iterative learning to the air-to-ground heterogeneous unmanned swarm system to achieve formation control of the leader UAV and encirclement control of the follower UAVs.

[0040] In this embodiment, learnable adaptive parameters are also introduced. These adaptive parameters are updated during each policy iteration; when the adaptive parameters do not reach the dual-policy switching boundary, a stable policy iteration is performed; when the adaptive parameters reach the dual-policy switching boundary, the system switches to performing the optimal policy iteration.

[0041] Therefore, by introducing additional stabilizing terms, adaptive parameters, and a phased strategy iteration mechanism, the reinforcement learning control strategy can start iterative learning without the need for a preset initial stable control strategy. Furthermore, by adjusting the poles of the closed-loop system through additional stabilizing terms, the closed-loop system in the early stage of iteration satisfies the Hurwitz stability condition, thereby reducing the algorithm's dependence on prior stable control strategies and accurate dynamic models, and improving its applicability in air-ground heterogeneous unmanned swarm systems with unknown models.

[0042] To enable those skilled in the art to more clearly understand the technical solution of the present invention, the formation control method for air-to-ground cluster systems oriented towards cooperative target search in complex environments, provided by the embodiments of the present invention, is described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0043] Figure 1 The flowchart illustrates the formation control method for air-to-ground cluster systems in complex environments, provided by an embodiment of the present invention. The specific implementation steps of the method are as follows: Step 1: Establish an air-to-ground unmanned cluster system model.

[0044] Establish dynamic models for the leader drone, the follower drone, the virtual leader, and the desired formation, and construct a globally unified description model based on the above models. (1) The first in the constructed air-to-ground unmanned cluster system i The dynamic model of the leader drone is as follows:

[0045] in, For leaders to assemble drones, The leader's total number of drones, which is also the largest drone number. for The derivative; and They represent the first i The state vector and control input of the leader drone, and They represent the first i The three-dimensional position and attitude of the leader drone, among which This is the roll angle. The pitch angle, Yaw angle; and They are respectively and The derivative of . and They represent the firsti The control inputs for each subsystem of the leader drone, among which... and These are the desired translational speed and yaw rate, respectively. and They represent the first i The system matrix and input matrix of the leader drone. and They respectively represent the first i The parameter matrices related to the three-dimensional position and yaw angle subsystem of the leader drone are as follows:

[0046] in, and Indicates the first i The aerodynamic drag coefficient of a leader drone Indicates the first i The quality of a leader's drone g This is the acceleration due to gravity. and ,in, The length of the drone's arm. and Let be the moment of inertia. and For the first i The control gain of the leader drone autopilot.

[0047] (2) The first unmanned cluster system constructed in the air-to-ground region The dynamic model of the follower autonomous vehicle is as follows:

[0048] in, To gather the following driverless cars, The total number of follower drones, and the number of the follower drones following the leader drone. and The first The state vector and control input of a follower autonomous vehicle. and They represent the first The position and velocity of a follower autonomous vehicle in a two-dimensional plane. and They are respectively and The derivative of . and The first The control inputs for each subsystem of the follower drone. and The first The system matrix and input matrix of a follower autonomous vehicle. and They respectively represent the first The parameter matrix related to the two-dimensional position subsystem of a follower autonomous vehicle, where... and , For the first The quality of a follower driverless car.

[0049] (3) The dynamic model of the virtual leader in the constructed air-to-ground unmanned cluster system is as follows:

[0050] in, and These represent the system state and bounded control inputs of the virtual leader, respectively. and These are the system matrix and input matrix of the virtual leader, respectively.

[0051] (4) The desired formation in the constructed air-to-ground unmanned swarm system is as follows:

[0052] in, For the first i The leader drone and the first j The expected 3D position deviation and expected yaw angle deviation between the leader drones. and The first i The leader drone, the first j The expected 3D position deviation and expected yaw angle deviation between the leader drone and the virtual leader.

[0053] (5) Based on the aforementioned leader drone dynamics model, follower drone dynamics model, virtual leader dynamics model, and desired formation, a globally unified description model is constructed, specifically expressed as follows:

[0054] in, and These represent the virtual leaders. The system state, control inputs, and system outputs of each subsystem. , and These represent the virtual leaders. The system matrix, input matrix, and output matrix of each subsystem. , These correspond to the three-dimensional position and yaw angle, respectively. and Indicates the first in the air-to-ground unmanned aerial vehicle (UAV) swarm system i The leader drone The status, control inputs, and outputs of each subsystem. and They are respectively with the first i The dynamics matrix that matches the leader drone model. . and Indicates the first in the air-to-ground unmanned aerial vehicle (UAV) swarm system The first follower driverless car The status, control inputs, and outputs of each subsystem. and They are respectively with the first The dynamics matrix that matches a follower autonomous vehicle model. . For the first i A leader drone and j Among the leader drones in the first Expected position deviation or expected yaw angle deviation in each dimension; and The first i The leader drone, the first j Between the leader drone and the virtual leader in the [number]th [period] Dimensional deviation of expected position or expected yaw angle; .

[0055] Step 2: Construct a two-layer distributed observer.

[0056] A distributed observer is constructed, comprising a formation layer observer and an encirclement layer observer, wherein the formation layer observer is used to estimate the formation trajectory reference generated by the virtual leader, and the encirclement layer observer is used to estimate the encirclement reference state within the convex hull formed by the leader UAV.

[0057] (1) A two-layer distributed observer is constructed using symbolic functions. The formation layer observer is as follows:

[0058] in, and These represent the first [unit / item] used in the air-to-ground unmanned swarm system. i Each leader drone provides state and control inputs to a formation layer observer for distributed state estimation. For the derivative of the state of the formation layer observer, This refers to the neighboring formation following error. For the first i A collection of neighboring intelligent agents of a leader drone. For the first i The leader drone and the first j The adjacency weight of the communication topology between the leader drones, if If a communication connection exists, then Greater than zero, otherwise It equals zero; For virtual leadership and the i The connection weight between the leader drones is determined if a communication connection exists between them. Greater than zero, otherwise It equals zero. and It is a positive gain constant. For symbolic functions, For the control gain of the formation layer observer, and By positive definite symmetric matrix Determined; the positive definite symmetric matrix By solving the system of linear matrix inequalities get, , It is the information transmission matrix corresponding to the communication network topology between the virtual leader and the leader drone in the air-to-ground unmanned swarm system.

[0059] (2) Construct a confinement layer observer based on the sign function. The constructed confinement observer is as follows:

[0060] in, and These represent the first [unit / item] used in the air-to-ground unmanned swarm system. Each follower autonomous vehicle provides state and control inputs to an enclosing layer observer for distributed state estimation. The derivative of the state of the enclosed layer observer. This represents the neighborhood encirclement and following error. For the first A collection of neighboring intelligent agents of a follower autonomous vehicle. For the first The follower driverless car and the first The adjacency weight of the communication topology between each follower autonomous vehicle, if If a communication connection exists, then Greater than zero, otherwise It equals zero; For the first i The leader drone and the first The connection weight between each follower autonomous vehicle is determined if a communication connection exists between them. Greater than zero, otherwise It equals zero. and It is a positive gain constant. For symbolic functions, For the control gain of the enclosing layer observer, and By positive definite symmetric matrix Determined; the positive definite symmetric matrix This can be achieved by solving linear matrix inequality equations. get, , It is the information transmission matrix corresponding to the communication network topology between the leader drone and the follower drones in an air-to-ground unmanned swarm system.

[0061] Step 3: Based on the distributed observer states obtained in Step 2 and the air-to-ground unmanned swarm system model established in Step 1, construct augmented systems for the formation-layer UAV system and the encirclement-layer unmanned vehicle system, respectively, and design the corresponding distributed controllers and control gain matrices based on state feedback control theory. and ; (1) The augmented system dynamics model of the formation-layer UAV system is as follows:

[0062] in, For formation tracking error, To integrate the first air-to-ground unmanned swarm system i The leader drone The state of each subsystem and the matching formation layer observer state The constructed augmented vector, This is the control input for the augmented subsystem. , and These are the system matrix, input matrix, and output matrix of the augmented system, respectively. This is an augmented matrix related to the observer control input matrix. As an input term related to the neighborhood formation following error, when the formation observer converges, It converges asymptotically to zero. The formation control gain matrix to be learned. This is the control gain matrix related to the feedforward term, i.e., the observer state. This is the control gain matrix related to the feedback term, i.e., the leader drone state.

[0063] The model-based optimal formation controller is designed as follows:

[0064] in, and This is the weight matrix. and . For optimal formation controller, This represents the optimal feedback gain matrix. The superscript " " indicates optimal. It is obtained by solving the algebraic Riccati equation for the discount factor, which is expressed as:

[0065] in, This is a discount factor used to ensure the convergence of the formation controller.

[0066] (2) The augmented system dynamics model of the enclosed layer unmanned vehicle system is as follows:

[0067] in, For formation tracking error, To integrate the first air-to-ground unmanned swarm system The first driverless car The state of each subsystem and the matching enclosing layer observer state The constructed augmented vector, This is the control input for the augmented subsystem. , and These are the system matrix, input matrix, and output matrix of the augmented system, respectively. This is an augmented matrix related to the observer control input matrix. As an input term related to the neighborhood formation following error, when the enclosing layer observer converges, It will asymptotically converge to zero. The encircling control gain matrix to be learned. This is the control gain matrix related to the feedforward term, i.e., the observer state. This is the control gain matrix related to the feedback term, i.e., the state of the follower autonomous vehicle.

[0068] The model-based optimal encirclement controller is designed as follows:

[0069] in, and This is the weight matrix. , and . For optimal encirclement controller, This represents the optimal feedback gain matrix. It can be obtained by solving the algebraic Riccati equation for the discount factor, which is expressed as:

[0070] in, This is a discount factor used to ensure the convergence of the encirclement controller.

[0071] Step 4: Use a dual-strategy iterative reinforcement learning algorithm to iteratively optimize the control strategy.

[0072] For the optimal formation controller at the formation layer and the optimal encirclement controller at the encirclement layer obtained in step 3, a dual-strategy iterative reinforcement learning algorithm is used to iteratively optimize the control strategy. This step applies initial control inputs, including piecewise constant and continuous excitations, to the air-to-ground heterogeneous unmanned swarm system and collects sufficiently rich system operation data. A data storage matrix is ​​constructed using the integral of the collected system operation data to avoid constructing a high-dimensional data matrix using the Kronecker product. Furthermore, a smaller non-singular data matrix is ​​constructed according to the data index. To eliminate the dependence on the initial stable control strategy, an additional stability term at the formation layer is introduced during strategy iteration. and the additional stability term of the enclosing layer And set the corresponding adaptive update parameters. and This ensures that the closed-loop system matrix maintains the Hurwitz stability condition during the initial learning phase, thereby guaranteeing that the generalized Hillwest transpose data equation is solvable and can be iteratively updated. When the adaptive parameters... and Once the corresponding switching conditions are met, the learning algorithm switches from the stable policy iteration phase to the optimal policy iteration learning phase. At this point, the algorithm is equivalent to the Kleinman policy iteration and converges to the optimal solution through further iterations. Finally, the learned optimal control policy is applied to the air-ground heterogeneous unmanned swarm system to achieve optimal formation encirclement and collaborative control of the air-ground heterogeneous unmanned swarm system.

[0073] In this embodiment of the invention, the dual-policy iterative reinforcement learning algorithm includes the following steps: S401: Collect system operation data and construct a basic data storage matrix.

[0074] An initial control input containing piecewise constant continuous excitation is applied to the air-to-ground swarm unmanned system. The data sampling time series for the leader drone and the follower drone are set as follows: and , where subscript t This represents the sampling sequence number; for the leader drone, the sampling sequence number is 0- For follower autonomous vehicles, the sampling sequence number ranges from 0 to... The sampling step size is The data lengths collected by the leader drone and the follower drone were respectively... And, where the number of data samples and number of data samples ,in, and Representing the system state respectively and The corresponding dimension and These represent control inputs respectively. and The corresponding dimension.

[0075] The collected operational data of the air-to-ground cluster unmanned system are integrated, and a data storage matrix excluding the Kronecker product is constructed. , , and , is represented as:

[0076] in, and The data stored in the middle contains the status of the leader drone during the operation of the air-to-ground unmanned swarm augmentation system. Related system operation data and follower autonomous vehicle status Related system operation data; and The data stored in the middle contains the data input from the leader drone during the operation of the air-to-ground unmanned swarm augmentation system. Related system operation data and input from the follower autonomous vehicle Related system operation data. and They represent the sub-intervals respectively. within, no. i The leader drone The augmentation subsystem and the first The first follower driverless car System status data collected by each augmented subsystem; and They represent the sub-intervals respectively. within, no. i The leader drone The augmentation subsystem and the first The first follower driverless car The augmentation subsystem collects system input data. This is the integration step size.

[0077] To eliminate the algorithm's dependence on the initial stable control strategy during the learning process, an additional stabilization term is introduced. and , is represented as:

[0078] in, and These are the first air-to-ground unmanned swarm systems. i The leader drone The augmentation subsystem and the first The first follower driverless car Additional stabilizing terms for each augmented subsystem; and These represent the switching boundary parameters during the iteration process of the corresponding augmented subsystem. The switching boundary parameters are learned through step S403 and remain fixed in subsequent iterations. and These represent the adaptive parameters during the iterative process of the corresponding augmented subsystem. and Represents the identity matrix, subscript It is the identity matrix order It is the identity matrix The order of; As can be seen from the above relationships, there is a corresponding conversion relationship between the additional stabilization term, the switching boundary parameter, and the adaptive parameter. The additional stabilization term participates in the data storage matrix. and The construction of the non-singular data matrix is ​​influenced by the fact that the non-singular data matrix is ​​the data basis for the generalized Hillwest transpose matrix equation. Therefore, when the additional stabilization term changes, the data within the non-singular data matrix changes accordingly, thus affecting the solution matrix and its positive definiteness judgment result. This invention achieves adaptive stabilization startup of the reinforcement learning algorithm without needing to know the initial stable control strategy by having the switching boundary parameters take values ​​according to a preset monotonically increasing sequence and solving the corresponding matrix equation after each value. When the solution matrix satisfies the positive definiteness condition, the current switching boundary parameters are determined as the switching condition for dual-strategy iterative optimization.

[0079] The update laws for the adaptive parameters are as follows:

[0080] Among them, superscript k Indicates the number of iterations. and To iteratively update the step size, and These are the first air-to-ground unmanned swarm systems. i The leader drone The augmentation subsystem in the first k+ The control gain matrix in the first iteration and the first iteration k The solution matrix of the Riccati equation in the next iteration; and The first The first follower driverless car The augmentation subsystem in the first k+ The control gain matrix in the first iteration and the first iteration k The solution matrix of the Riccati equation in the next iteration. and Let represent the minimum and maximum eigenvalues ​​of the matrix, respectively. and This is the auxiliary weight matrix.

[0081] Step 402: Construct the state differential data storage matrix and the non-singular data matrix.

[0082] Using the additional stability term designed in step S401 and The collected system operation data is preprocessed, and a data storage matrix for the system state differential is constructed. and , is represented as:

[0083] in, and The data stored in the two locations are the differential states of the leader drone and the air-to-ground drone during the operation of the air-to-ground drone swarm augmentation system. The state differential of the following autonomous vehicles Relevant system operation data. and They respectively represent the subintervals within, no. i The leader drone The augmentation subsystem and the first The first follower driverless car The system state differential data collected by each augmented subsystem. Subsequently, the data storage matrix constructed in step S401 and this step is used. , , , , and Select a set of indexes as and Construct a data matrix with a relatively small data volume. , , , and the k The state differential related data matrix for the next iteration and and the corresponding data matrix constructed and Satisfying the non-singular condition, where, and These represent the sets of indices used to select formation layer data and encirclement layer data, respectively. and They represent the first i The dimensions of state-related data items and input-related data items in each formation-layer augmented subsystem; and They represent the first The dimension of state-related data items and input-related data items in each enclosed layer augmented subsystem. and They represent by index set The selected number r 1 set of formation layer status data and input data; and They represent by index set The selected number r 2. Combine the surrounding layer status data and input data. and They represent the first k During the nth iteration, the... r Group 1 and the r The differential correlation data columns of the formation layer and encirclement layer states were calculated from the two sets of state data. and By using the above-described data construction method without Kronecker product, the computational and storage overhead caused by matrix dimensionality expansion can be reduced.

[0084] S403: Determine the boundary parameters for switching between dual strategies.

[0085] Let the initial control gain be... And learn the switching boundary parameters based on the following data-driven generalized Hillwest transpose matrix equation. and :

[0086] in, and Let be the parameter matrix to be learned. and For small positive constants, , , and .

[0087] Subsequently, the learned solution matrix of the Riccati equation was analyzed. Perform positive definiteness verification: if the solution matrix If the positive definite condition is met, it indicates that the current adaptive parameters are... This ensures that the initial closed-loop system satisfies the Hurwitz stability condition, and the current... The solution matrix is ​​determined to be a valid switching boundary parameter; if the solution matrix is ​​determined to be a valid switching boundary parameter. If the positive definiteness condition is not met, then it follows the preset monotonically increasing sequence. Gradually increase The value of is then used to resolve the above generalized Hillwest transpose matrix equation until the solution matrix is ​​found. The positive definiteness condition is met. Following the same method, the effective switching boundary parameters for the follower-type autonomous vehicle system can be obtained. .

[0088] The adaptive parameter is compared with the switching boundary parameter. When the adaptive parameter does not reach the corresponding switching boundary parameter, the stable strategy iteration is executed and the process proceeds to step S404. When the adaptive parameter reaches the corresponding switching boundary parameter, the process switches to the optimal strategy iteration and proceeds to step S405.

[0089] S404: Perform stable policy iterations and learn the initial stable control gain matrix. If the adaptive parameters do not reach the corresponding switching boundary parameters, execute the stable strategy iteration.

[0090] Specifically, when and ,Right now and hour, and ,in, and These represent the iteration counts corresponding to the achievement of the respective switching boundary parameters. Based on the system operation data in the data storage unit, the following data-driven generalized Hillwest transpose matrix equation is iteratively solved:

[0091]

[0092] in, and Let be the parameter matrix to be learned. , and , and .

[0093] Subsequently, the control gain matrix of the UAV is updated using the solved parameters. and the control gain matrix of autonomous vehicles This enables the air-to-ground unmanned swarm system to learn the control gain matrix that meets stability requirements.

[0094] S405: Perform optimal policy iteration and learn the optimal control gain matrix. Once the adaptive parameters reach the corresponding switching boundary parameters, the optimal strategy iteration phase begins. Specifically, when... and ,Right now and back, and Based on the system operation data in the data storage unit, the following data-driven generalized Hillwest transpose matrix equation is solved iteratively:

[0095]

[0096] Subsequently, the control gain matrix of the leader drone is updated using the solved parameters. and the control gain matrix of autonomous vehicles This enables the air-to-ground unmanned swarm system to learn the optimal control gain matrix.

[0097] S406: Determine whether the control strategy iteration is complete.

[0098] The algorithm determines whether the difference between the control gain matrix obtained in the current iteration and the control gain matrix obtained in the previous iteration is less than a preset convergence threshold. If so, it stops iterating and obtains the optimal control gain matrices for the leader UAV and follower UAVs in the air-to-ground swarm system. Simultaneously, it obtains the optimal formation controller at the formation layer and the optimal encirclement controller at the encirclement layer. Specifically: judge and Whether it is true or not, among which, A preset constant that is greater than zero; ①If and If the condition is met, the iteration stops. At this point, the control gain of the leader drone in the air-to-ground drone swarm system is... The corresponding controller is The control gain of the follower autonomous vehicle is The corresponding controller is Proceed to step S407; ②If and If not, then Then, the iteration continues in S405; S407: Obtain the optimal control gain matrix of the leader drone in an air-to-ground unmanned swarm system. Optimal control gain matrix for follower autonomous vehicles At the same time, the optimal formation controller at the formation level is obtained. and the optimal encirclement controller of the encirclement layer .

[0099] This concludes the process.

[0100] Furthermore, the air-to-ground cluster system formation control method for collaborative target search in complex environments according to the embodiments of this application can all be implemented by a single computer device. Figure 3 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application. Figure 3 As shown, the device may include a processor 201 and a memory 202 storing computer program instructions.

[0101] Specifically, the processor 201 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0102] The memory 202 may include a large-capacity memory for data or instructions. For example, and not limitingly, the memory 202 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 202 may include removable or non-removable (or fixed) media. Where appropriate, the memory 202 may be internal or external to a data processing device. In a particular embodiment, the memory 202 is non-volatile memory. In a particular embodiment, the memory 202 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0103] The memory 202 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 201.

[0104] The processor 201 reads and executes computer program instructions stored in the memory 202 to implement any of the air-to-ground cluster system formation control methods for collaborative target search in complex environments as described in the above embodiments.

[0105] In some embodiments, the computer device may further include a communication interface 203 and a bus 200. For example, Figure 3 As shown, the processor 201, memory 202, and communication interface 203 are connected through bus 200 and complete communication with each other.

[0106] The communication interface 203 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 203 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0107] Bus 200 includes hardware, software, or both, that couples components of a computer device together. Bus 200 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 200 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 200 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnection is contemplated herein.

[0108] The computer device can execute the air-to-ground cluster system formation control method for collaborative target search in complex environments as described in the embodiments of this application, thereby realizing the air-to-ground cluster system formation control described in this application.

[0109] Furthermore, in conjunction with the air-to-ground swarm system formation control method for cooperative target search in complex environments described in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the air-to-ground swarm system formation control methods for cooperative target search in complex environments described in the above embodiments.

[0110] It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. In addition, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of steps / components can be combined into new steps / components to achieve the purpose of this invention.

[0111] The specific embodiments described above only illustrate the design principles of the present invention. The shapes and names of the components in this description may differ and are not limited. Therefore, those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments; and these modifications and substitutions do not depart from the inventive spirit and technical solutions of the present invention, and should all fall within the protection scope of the present invention.

Claims

1. A formation control method for an air-to-ground swarm system for cooperative target search in complex environments, characterized in that, In the air-to-ground swarm system, the virtual leader issues a global search command. The leader drones form an aerial observation array according to the desired formation to conduct aerial searches in complex environments. When a suspected target is detected, the aerial observation array moves as a whole, driving the convex hull area to synchronously move towards the target area. The follower drones maintain communication and linkage with the leader drones, and their movement range is always constrained within the convex hull area. They move with the convex hull to approach the target, realizing the formation, encirclement, and collaborative control of the air-to-ground swarm. This method employs a dual-strategy iterative reinforcement learning approach to optimize the control strategy, including: Collect system operation data under piecewise constant continuous excitation, and construct a data matrix by combining the system operation data with additional stability terms; The additional stabilization term has a corresponding conversion relationship with the dual-strategy switching boundary parameters, causing the data matrix to change with the switching boundary parameters, thereby achieving strategy learning initialization without a preset model-based initial stabilization strategy; a generalized Hill West transpose matrix equation based on data is constructed and substituted into the data matrix for solving, determining the minimum switching boundary parameters that make the matrix equation produce a positive definite solution matrix, and using this as the dual-strategy switching boundary; based on the dual-strategy switching boundary, stabilization strategy iteration and optimal strategy iteration are performed in stages, and the learned optimal control strategy is applied to the air-to-ground cluster system.

2. The method as described in claim 1, characterized in that, The specific process of constructing and iteratively optimizing the formation control strategy of the air-to-ground cluster system is as follows: Step 1: Establish dynamic models for the virtual leader, leader drone, and follower drone, and combine them with the desired formation to construct a globally unified description model of the air-to-ground cluster system. Step 2: Construct a two-layer distributed observer, in which the formation layer observer is used to estimate the formation reference trajectory information generated by the virtual leader and the desired formation, and the encirclement layer observer is used to estimate the encirclement reference state information located within the convex hull formed by the leader; Step 3: Combining the global unified description model of the cluster system and the two-layer distributed observer, construct the formation layer augmentation system and the encirclement layer augmentation system respectively, and design the corresponding distributed state feedback controller accordingly; Step 4: For the formation layer controller and the encirclement layer controller, a dual-strategy iterative reinforcement learning algorithm is used to learn and optimize the control strategy, specifically including: Step 41: Apply piecewise constant and continuous excitation input to the air-to-ground cluster system, collect system operation data and perform integration processing to construct a first data storage matrix without Kronecker product form; Step 42: Introduce additional stabilization terms to preprocess the collected system operation data to construct a second data storage matrix for the state differential of the air-to-ground cluster system; select a set of linearly independent column vector data from the first data storage matrix and the second data matrix according to the data index to construct a small-volume non-singular data matrix. Step 43: Construct the generalized Hillwest transpose matrix equation based on the non-singular data matrix and solve the Riccati equation solution matrix; gradually increase the switching boundary parameter according to a preset monotonically increasing sequence, wherein the changed switching boundary parameter will affect the value of the additional stabilizing term, thereby changing the data in the non-singular data matrix; reconstruct the non-singular data matrix based on the updated additional stabilizing term, and repeatedly solve the generalized Hillwest transpose matrix equation using the new non-singular data matrix until the switching boundary parameter that makes the Riccati solution matrix satisfy the positive definite condition is obtained, and it is determined as the dual-policy switching boundary, so that the non-singular data matrix constructed under the corresponding additional stabilizing term satisfies the stability policy learning initialization requirements of the reinforcement learning algorithm; Step 44: First-layer policy iteration: Learning a stable control gain matrix; When the adaptive parameters do not reach the switching boundary parameters, the first-layer strategy iterative learning is performed to learn the stable control gain matrix: the current additional stable term is calculated based on the switching boundary parameters and the adaptive parameters of the current iteration round; a non-singular data matrix is ​​constructed based on the current additional stable term; and the control gain matrix and the solution matrix of the Riccati equation are solved by substituting the generalized Hillwest transpose matrix equation to learn the stable control gain matrix; and the adaptive parameters of the next iteration round are updated and calculated using the solved control gain matrix and the solution matrix of the Riccati equation. Step 45: Second-layer policy iteration: Learning the optimal control gain matrix; When the adaptive parameter reaches the switching boundary parameter, switch to the second-level strategy to iteratively learn the optimal control gain matrix: set the current additional stabilizing term to zero, construct a non-singular data matrix based on the zeroed additional stabilizing term, and substitute it into the generalized Hillwest transpose matrix equation and solve iteratively to learn the optimal control gain matrix. Step 5: Based on the optimal control gain matrix obtained from the final learning, obtain the optimal control strategy and apply the optimal control strategy to the air-ground swarm system to achieve optimal formation and encirclement cooperative control of the air-ground heterogeneous unmanned swarm of UAVs and unmanned vehicles, and complete the cooperative target search in complex environments.

3. The method as described in claim 2, characterized in that, In step 42, an additional stability term is introduced to preprocess the collected system operation data and construct a second data storage matrix for the state differential of the air-to-ground cluster system, specifically including: Superscript of the defined parameter Indicates the first The next iteration; the subscript of the parameter In the air-to-ground cluster system, the first i The leader drone's first An augmentation subsystem, subscript Indicates the first The first follower driverless car One augmentation subsystem; and These are the corresponding augmentation subsystems. and Additional stabilizing terms; The data sampling time series for the leader drone and the follower drone are set as follows: and ; Using additional stabilizing terms and The collected system operation data is preprocessed by integration to construct a second data storage matrix for the state differential of the air-to-ground cluster system. and ,in: in, The data stored contains the differential state of the leader drone during the operation of the air-to-ground swarm augmentation system. Related system operation data, It is the first i The leader drone's first Augmented vectors of augmented subsystems yes The differential, The parentheses after the last one Indicates the sampling time. The subscript indicates the sequence number in the sampling time series; The data stored contains the state differentials between the air-to-ground cluster augmentation system and the follower unmanned vehicle during operation. Related system operation data; It is the first The first follower driverless car Augmented vectors of augmented subsystems yes The differential; This is the integration step size.

4. The method as described in claim 2, characterized in that, Step 43 specifically includes: The switching boundary parameters are set to the initial values ​​of a preset monotonically increasing sequence, and the initial control gain matrix is ​​set to zero. Based on the corresponding conversion relationship between the additional stable terms and the switching boundary parameters, the current additional stable terms of the leader UAV augmentation subsystem and the follower UAV augmentation subsystem are determined respectively. Based on the current additional stable terms, the second data storage matrix is ​​constructed using the construction method in step 42, and combined with the first data storage matrix, the current non-singular data matrix is ​​obtained. Substitute the current non-singular data matrix into the generalized Hillwest transpose matrix equation to solve the matrix equation solution matrix containing the Riccati equation solution matrix and the control gain matrix; The positive definiteness of the solution matrix of the Riccati equation is then verified. If the solution matrix is ​​positive definite, the current switching boundary parameters ensure that the system satisfies the Hurwitz stability condition, and the current switching boundary parameters are determined as the dual-strategy switching boundary. If the solution matrix is ​​not positive definite, the value of the switching boundary parameters is gradually increased according to a preset monotonically increasing sequence, and the current additional stable term and the current non-singular data matrix are recalculated. The matrix equation is then repeatedly solved until the switching boundary parameters that make the solution matrix of the Riccati equation positive definite are obtained. , to use it as the boundary for switching between the two strategies; The conversion relationship between the additional stability term and the switching boundary parameter is as follows: ； Among them, superscript Indicates the first Next iteration; subscript In the air-to-ground cluster system, the first i The leader drone's first An augmentation subsystem; subscript Indicates the first The first follower driverless car One augmentation subsystem; and These are the corresponding augmentation subsystems. and Additional stabilizing terms; and They represent the corresponding augmentation subsystems. and Switching boundary parameters; and They represent the corresponding augmented subsystems. and Adaptive parameters can be learned and optimized during the iteration process; and Represents the identity matrix, subscript It is the identity matrix The order of represents the corresponding augmented subsystem. The state dimension; It is the identity matrix The order of represents the corresponding augmented subsystem. The state dimension; Wherein, the adaptive parameter and The update law is: in, and To update the step size iteratively; and These are the first air-to-ground cluster systems. The leader drone's first During the iterative learning process of the augmented subsystem, the first... The control gain matrix of the second iteration and the first iteration The solution matrix of the Riccati equation in the next iteration; and The first The first follower driverless car During the iterative learning process of the augmented subsystem, the first... The control gain matrix of the second iteration and the first iteration The solution matrix of the Riccati equation in the next iteration. and Let represent the minimum and maximum eigenvalues ​​of the matrix, respectively. and For auxiliary weight matrix; and The first The leader drone's first The augmentation subsystem and the first The first follower driverless car The state weight matrix corresponding to each augmented subsystem and The first The leader drone's first The augmentation subsystem and the first The first follower driverless car The control weight matrix corresponding to each augmented subsystem; For augmentation subsystem The corresponding output weight matrix, For augmented subsystems The corresponding output matrix; For augmented subsystems The corresponding output weight matrix, For augmentation subsystem The corresponding output matrix; and This is an intermediate quantity.

5. The method as described in claim 1, characterized in that, Step one specifically includes: (1) Constructing the first air-to-ground cluster system i Dynamics model of a leader drone: in, For leaders to assemble drones, This represents the total number of drones belonging to the leader, and is also the largest drone number. and The first i The state vector and control inputs of the leader drone; and These are the corresponding system matrix and input matrix, respectively; for The derivative; (2) Constructing the first air-to-ground cluster system Dynamics model of a follower autonomous vehicle: in, To gather the following driverless cars, The total number of follower drones, followed by the leader drone's number; and The first The state vector and control input of a follower autonomous vehicle; and These are the corresponding system matrix and input matrix, respectively; (3) Construct a dynamic model of the virtual leader in the air-to-ground cluster system: in, and These represent the system state and bounded control input of the virtual leader, respectively. and These are the system matrix and input matrix of the virtual leader, respectively; (4) Based on the task requirements, construct the desired formation in the air-to-ground cluster system: in, For the first i The and the first j The expected state deviation between the leader drones; and The first i The and the first j The expected state deviation between the leader drone and the virtual leader; (5) Based on the dynamic models of the leader drone, follower drones, and virtual leader, the desired formation is combined to construct a globally unified description model, including the following subsystem forms: in, and These represent the virtual leaders. The system state, control inputs, and system outputs of each subsystem. , and These represent the virtual leaders. The system matrix, input matrix, and output matrix of each subsystem, with subscripts. ,in These represent the three-dimensional position and yaw angle dimensions, respectively. and In the air-to-ground cluster system, the first The leader drone The status, control inputs, and outputs of each subsystem and These are the dynamic matrices that match the leader drone dynamics model; and In the air-to-ground cluster system, the first The first follower driverless car The status, control inputs, and outputs of each subsystem and These are the dynamic matrices that match the dynamics model of the follower autonomous vehicle; For the first and Among the leaders of the drones, the first Dimensional state deviation, and The first i and j Between the leader drone and the virtual leader Dimensional state deviation; , These represent the three-dimensional position and yaw angle, respectively.

6. The method as described in claim 5, characterized in that, In step two, a two-layer distributed observer is constructed based on the symbolic function. The two-layer distributed observer includes a formation layer observer and an encirclement layer observer. The formation layer observer is used to estimate the state of the leader UAV in an air-to-ground heterogeneous unmanned swarm system. The formation layer observer is represented as follows: in, and These represent the first [unit / item] used in the air-to-ground cluster system. Each leader drone provides state and control inputs to a formation layer observer with distributed state estimation; For formation layer observation status The derivative; and It is a positive gain constant. It is a symbolic function; For the control gain of the formation layer observer, and By positive definite symmetric matrix Determined, wherein the positive definite symmetric matrix By solving linear matrix inequality equations get; , It is the information transmission matrix corresponding to the communication network topology diagram between the virtual leader and the leader drone in an air-to-ground unmanned swarm system. Represents the smallest eigenvalue of the matrix; Indicates the first i Neighborhood formation following error of a leader drone; For Leaders Drones A collection of neighboring intelligent agents; For the first The leader drone and the first The adjacency weight of the communication topology between the leader drones, if , When a communication connection exists, Greater than zero, otherwise It equals zero; For virtual leadership and leader drones The connection weight between them; if a communication connection exists between them, then... Greater than zero, otherwise It equals zero; The encirclement observer is used to perform distributed estimation of the state of follower unmanned vehicles in an air-to-ground heterogeneous unmanned swarm system. The encirclement layer observer is represented as follows: in, and These represent the first [unit / item] used in the air-to-ground unmanned swarm system. Each follower autonomous vehicle provides state and control inputs to an enclosing layer observer for distributed state estimation. Enclosure layer observer status The derivative; For the first The neighborhood encirclement and following error of a follower autonomous vehicle; For follower driverless cars A collection of neighboring intelligent agents; and It is a positive gain constant; It is a symbolic function; For the control gain of the enclosing layer observer, and By positive definite symmetric matrix Determined, wherein the symmetric matrix By solving linear matrix inequality equations get, , It is the information transmission matrix corresponding to the communication network topology between the leader drone and the follower drones in an air-to-ground unmanned swarm system. For the first and The adjacency weight of the communication topology between each follower autonomous vehicle, if and If a communication connection exists, then Greater than zero, otherwise It equals zero; To lead drones With follower driverless cars The connection weight between them; if a communication connection exists between them, then... Greater than zero, otherwise It equals zero.

7. The method as described in claim 2, characterized in that, In step four, the methods for determining whether the control strategy iteration optimization is complete include: If the difference between the control gain matrix obtained in the current iteration and the control gain matrix obtained in the previous iteration is less than a preset convergence threshold, then the control strategy iteration optimization is completed and the iteration is stopped. The optimal control gain matrix corresponding to the leader UAV and the follower UAV in the air-ground heterogeneous unmanned swarm system is obtained, and then the optimal formation controller and the optimal encirclement controller at the formation layer and encirclement layer are obtained.

8. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores programs or instructions that can run on the processor, and when the programs or instructions are executed by the processor, implement the steps of the air-to-ground cluster system formation control method for cooperative target search in complex environments as described in any one of claims 1 to 7.

9. A readable storage medium, characterized in that, It stores programs or instructions, which, when executed by a processor, implement the steps of the air-to-ground cluster system formation control method for collaborative target search in complex environments as described in any one of claims 1 to 7.

Images