Composite game reinforcement learning method and device for quadrotor unmanned aerial vehicle cluster

Abstract

The application discloses a compound game reinforcement learning method and device for a quadrotor unmanned aerial vehicle cluster, and relates to the technical field of unmanned aerial vehicle cluster control. The method comprises the following steps: directly approaching value evaluation and cooperative strategy in the "state-control input" data space of cluster flight by using the "evaluation-execution" network of the unmanned aerial vehicle, avoiding repeatedly solving the high-dimensional coupled HJ equation set under the strong coupling nonlinear dynamics of the unmanned aerial vehicle, and thus significantly reducing the modeling accuracy requirement and the calculation cost. By introducing a filtering mechanism with a "forgetting factor", the pose / velocity / relative formation error, control instruction, energy consumption and safety cost and other samples collected in the historical flight process of the cluster are subjected to weighted integral processing, a compound "evaluation" network error and a regression signal fusing current and historical information are constructed, and a limited incentive criterion that can be online tested is given at the information matrix level, so that the unmanned aerial vehicle cluster can still maintain effective learning under the condition that the continuous PE condition is not met.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) swarm control technology, and in particular to a composite game-theoretic reinforcement learning method and apparatus for quadrotor UAV swarms. Background Technology

[0002] In UAV swarm cooperative flight and autonomous combat missions, multiple UAVs simultaneously perform tasks such as formation keeping, trajectory planning, target coverage, and conflict avoidance under conditions of shared airspace and communication resources. Each UAV needs to achieve optimal overall performance through cooperative mechanisms, but inevitably competes with others in terms of energy consumption, space occupation, communication bandwidth, and task priority. Their interaction behavior can be naturally modeled as a continuous-time non-zero-sum game problem. Specifically, each UAV acts as a player in the game, adjusting its own control inputs (such as thrust, attitude angular velocity, or desired acceleration) to minimize its own long-term performance index function while satisfying flight dynamics constraints, communication constraints, and safe distance constraints. Meanwhile, the control strategies of other UAVs synchronously affect the state evolution of the swarm system and the individual cost functions.

[0003] Theoretically, the Nash equilibrium strategy of a continuous-time non-zero-sum game can be characterized by a set of coupled Hamilton-Jacobi (HJ) equations. For UAV swarm systems, the state variables in the HJ equations typically include high-dimensional continuous states such as the positions, velocities, and attitudes of multiple UAVs, exhibiting highly nonlinear dynamics and strong coupling relationships. As the number of UAVs increases, the dimensionality of the HJ equations grows exponentially, making it difficult to obtain analytical solutions for this type of partial differential equation system. Traditional methods based on grid discretization or numerical approximation face problems of extremely high computational complexity, difficulty in guaranteeing convergence and real-time performance. In applications such as multi-UAV formation control, distributed obstacle avoidance, and cooperative maneuvering, directly solving the HJ equations based on an exact model is practically infeasible in engineering. Reinforcement learning can handle continuous states and control spaces without explicitly solving high-dimensional HJ equations, providing an effective technical approach to alleviate the "curse of dimensionality" and approximating the Nash equilibrium strategy of a non-zero-sum game in UAV swarms online. It has been gradually applied to UAV swarm cooperative control and autonomous decision-making problems.

[0004] How to achieve stable convergence of the weights of the "evaluation-execution" reinforcement learning network in a non-zero-sum game of quadrotor UAV swarms without relying on persistent excitation (PE) conditions or excessive injection of probe noise, while ensuring the stability and control performance of the closed-loop flight system, has become one of the key engineering problems that urgently need to be solved in UAV swarm reinforcement learning technology. Summary of the Invention

[0005] To address the shortcomings of existing collaborative control methods for drone swarms, this invention provides a composite game-theoretic reinforcement learning method and apparatus for quadrotor drone swarms. The technical solution is as follows:

[0006] On the one hand, a composite game-theoretic reinforcement learning method for quadrotor drone swarms is provided. This method is implemented by a composite game-theoretic reinforcement learning device for quadrotor drone swarms, and includes:

[0007] S1. Based on the state variables, control inputs, and rigid body dynamics model of each quadrotor drone swarm, establish a system model of the quadrotor drone swarm.

[0008] S2. Based on the system model, define the value function of each quadcopter drone. Through the system model, value function, game definition, and Nash equilibrium conditions, construct the Hamilton-Jacobi partial differential equation system. By solving the partial differential equation system, establish the Nash feedback strategy for each quadcopter drone.

[0009] S3. Construct a reinforcement learning evaluation-execution network for each quadcopter drone. Construct the evaluation network based on the value function and the execution network based on the Nash feedback policy.

[0010] S4. Based on the historical state of the quadcopter drone cluster and the current execution network weights and weight matrix, construct regression vectors and cost functions for each quadcopter drone. Based on the regression vectors and cost functions, construct the composite evaluation error of the evaluation network.

[0011] S5. Based on the composite evaluation error, regression vector and integral filter containing forgetting factor, construct the filtered error energy and the filtered regression energy, and then construct the normalized objective function of the evaluation network.

[0012] S6. Based on the filtered regression energy, regression vector, reconstruction cost function and integral filter, construct the information matrix and information vector of each quadcopter UAV respectively. When the information matrix is ​​full rank, construct the time operator according to the information matrix, and construct the finite excitation condition according to the minimum eigenvalue of the information matrix of the historical regression vector within a finite time window.

[0013] S7. Based on the information matrix, information vector, and time operator, and with the principle of minimizing the normalized objective function of the evaluation network, the weights of the evaluation network are updated by gradient descent. Based on the updated weights of the evaluation network, the filtered regression energy, the integral filter, the regression vector, and the gradient of the basis function, the weights of the execution network are updated.

[0014] S8. Based on the updated weights of the evaluation network, the weights of the execution network, and the current state variables and control inputs of the quadcopter drone swarm, determine the approximate Nash feedback strategy and obtain the control commands for the quadcopter drone swarm.

[0015] On the other hand, a composite game-theoretic reinforcement learning device for quadrotor drone swarms is provided. This device is applied to a composite game-theoretic reinforcement learning method for quadrotor drone swarms, and includes:

[0016] Establishment unit, used to establish system model of quadrotor drone swarm based on state variables, control inputs and rigid body dynamics model of each quadrotor drone;

[0017] The solution unit is used to define the value function of each quadcopter UAV based on the system model. Through the system model, value function, game definition and Nash equilibrium conditions, the Hamilton-Jacobi partial differential equation system is constructed. By solving the partial differential equation system, the Nash feedback strategy of each quadcopter UAV is established.

[0018] The first building unit is used to construct an evaluation-execution network for reinforcement learning for each quadcopter drone. The evaluation network is constructed based on the value function, and the execution network is constructed based on the Nash feedback policy.

[0019] The second construction unit is used to construct regression vectors and cost functions for each quadrotor drone based on the historical state of the quadrotor drone swarm and the current execution network weights and weight matrices. Based on the regression vectors and cost functions, the composite evaluation error of the evaluation network is constructed.

[0020] The third building unit is used to construct the filtered error energy and the filtered regression energy based on the composite evaluation error, regression vector and integral filter containing forgetting factor, and then construct the normalized objective function of the evaluation network.

[0021] The fourth construction unit is used to construct the information matrix and information vector of each quadcopter UAV based on the filtered regression energy, regression vector, reconstruction cost function and integral filter. When the information matrix is ​​full rank, a time operator is constructed based on the information matrix, and finite excitation conditions are constructed based on the minimum eigenvalue of the information matrix of the historical regression vector within a finite time window.

[0022] The update unit is used to update the weights of the evaluation network by gradient descent based on the information matrix, information vector, and time operator, with the principle of minimizing the normalized objective function of the evaluation network. The update unit updates the weights of the execution network based on the updated weights of the evaluation network, the filtered regression energy, the integral filter, the regression vector, and the gradient of the basis function.

[0023] The control unit is used to determine an approximate Nash feedback strategy based on the updated weights of the evaluation network, the weights of the execution network, and the current state variables and control inputs of the quadcopter drone swarm, and to obtain control commands for the quadcopter drone swarm.

[0024] On the other hand, a composite game-theoretic reinforcement learning device for quadcopter drone swarms is provided. The composite game-theoretic reinforcement learning device for quadcopter drone swarms includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the methods described above for composite game-theoretic reinforcement learning methods for quadcopter drone swarms is implemented.

[0025] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods of composite game-theoretic reinforcement learning for quadcopter drone swarms.

[0026] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0027] This invention proposes a non-zero-sum game-theoretic composite reinforcement learning method for cooperative control of UAV swarms. It utilizes the "evaluation-execution" network of each UAV to directly approximate value assessment and cooperative strategies online within the "state-control input" data space of the swarm flight. This avoids repeated offline solutions to the high-dimensional coupled HJ equations under the strongly coupled nonlinear dynamics of UAVs, significantly reducing modeling accuracy requirements and computational overhead. By introducing a filtering mechanism with a "forgetting factor," weighted integration is performed on samples collected during the swarm's historical flight, including pose / velocity / relative formation errors, control commands, and energy consumption and safety costs. This constructs a composite "evaluation" network error and regression signal that integrates current and historical information. Furthermore, a finite incentive criterion that can be verified online is provided at the information matrix level, enabling the UAV swarm to maintain effective learning even when the continuous PE condition is not met. This avoids engineering risks such as formation jitter, increased energy consumption, and reduced collision avoidance margin caused by long-term injection of detection noise. Based on this, the "evaluation-execution" update law of composite reinforcement learning deeply integrates the current weight estimate with the filtered historical data, which can ensure that the network weights converge to the optimal neighborhood, thereby obtaining the value solution of the corresponding game problem, and enabling the control input of each UAV to converge to the Nash equilibrium strategy of the non-zero-sum game. While improving the accuracy of parameter identification and policy approximation, it also improves the formation maintenance, trajectory tracking and anti-disturbance performance of the cluster closed loop, and has good real-time performance, deployability and engineering application value. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of a composite game-theoretic reinforcement learning method for quadcopter drone swarms provided by an embodiment of the present invention;

[0030] Figure 2 This is a flowchart of a composite game-theoretic reinforcement learning method for quadcopter drone swarms provided by an embodiment of the present invention;

[0031] Figure 3 This is a block diagram of a composite game-theoretic reinforcement learning device for quadcopter drone swarms provided in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the structure of a composite game-theoretic reinforcement learning device for quadcopter drone swarms provided in an embodiment of the present invention. Detailed Implementation

[0033] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0034] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0035] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0036] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0037] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0038] This invention provides a composite game-theoretic reinforcement learning method for quadrotor drone swarms. This method can be implemented using a composite game-theoretic reinforcement learning device for quadrotor drone swarms, which can be a terminal or a server. Figure 1 The flowchart shown is for a composite game-theoretic reinforcement learning method for quadrotor drone swarms. Figure 2 The flowchart shown is a composite game-theoretic reinforcement learning method for quadrotor drone swarms. The processing flow of this method may include the following steps:

[0039] S1. Based on the state variables, control inputs, and rigid body dynamics model of each quadrotor drone swarm, establish a system model of the quadrotor drone swarm.

[0040] In one feasible implementation, for application scenarios such as quadcopter UAV swarm formation flight, cooperative search, airspace collision avoidance, and multi-task collaboration, the swarm is modeled as follows: each quadcopter UAV is treated as an independent controlled unit, and its position, velocity, attitude, angular velocity, and relative pose with neighboring UAVs are considered as system states; the thrust, attitude control quantities, or acceleration commands of each UAV are considered as control inputs. The overall system is represented as a continuous-time dynamic system with multiple input channels, where each input channel corresponds to a UAV or a control decision-making entity.

[0041] Optionally, the specific operation of S1 may include the following S11-S15:

[0042] S11. In the inertial coordinate system, construct the state variables of each quadcopter UAV based on its center of mass position, center of mass velocity, roll / pitch / yaw Euler angles, and body angular velocity.

[0043] In one feasible implementation, the drone swarm consists of N quadcopter drones, numbered as follows: In inertial coordinate system In the equation (1), the true physical state of the i-th quadcopter UAV is defined as follows:

[0044] (1)

[0045] in, Let m be the position of the center of mass of the i-th quadcopter UAV. The velocity of the center of mass is (m / s). The Euler angles (rad) represent roll / pitch / yaw angles. The angular velocity of the aircraft is rad / s.

[0046] S12. Based on the total thrust and three-axis control torque of each quadcopter drone, construct the control input for each quadcopter drone.

[0047] In one feasible implementation, the executable control input of the quadcopter UAV is shown in equation (2) below:

[0048] (2)

[0049] in, The total thrust is (N). The torque is the three-axis control torque (N·m).

[0050] S13. Construct a rigid body dynamics model for each quadrotor UAV based on its state variables, control inputs, translational disturbances, rotational disturbances, rotation matrix, and Euler angular rate mapping matrix.

[0051] In one feasible implementation, let the mass of the i-th quadcopter UAV be... (kg), inertia matrix is (kg·m) 2 The acceleration due to gravity is g (m / s²). 2 Then the true rigid body dynamics of each quadcopter UAV can be written as follows (3):

[0052] (3)

[0053] in, and These represent translational / rotational disturbances caused by wind disturbance, aerodynamics, and modeling errors, respectively. Let be the unit vector in the vertical direction of the inertial frame. Rotation matrix. Take the ZXY sequence (yaw-pitch-roll) as shown in equation (4):

[0054] (4)

[0055] To reflect angular velocity and Geometric relationships, Euler angular rate mapping matrix Take the standard form, as shown in equation (5):

[0056] (5)

[0057] S14. Stack the state variables of all quadcopter drones to obtain the state variables of the quadcopter drone cluster. Stack the control inputs of all quadcopter drones to obtain the control inputs of the quadcopter drone cluster.

[0058] According to formula (1), the full state stacking of the quadcopter drone swarm is given by formula (6-1):

[0059] (6-1)

[0060] in, This indicates the compact set in which the quadcopter drone swarm is located.

[0061] According to formula (2), the total input of the quadcopter drone swarm is stacked as formula (6-2):

[0062] (6-2)

[0063] S15. Based on the state variables, control inputs, and rigid body dynamics model of each quadrotor UAV swarm, the system model of the quadrotor UAV swarm is expressed in affine form as shown in the following equation (7):

[0064] (7)

[0065] in, The state variables represent the state variables of a quadcopter drone swarm. Let N represent the control input for the i-th quadrotor UAV, where i represents the i-th quadrotor UAV and N represents the number of quadrotor UAVs. Indicates the drift state. This indicates that the input is dynamic.

[0066] Specifically, Describes the situation at zero control input (u i When =0), the autonomous evolution trend of the state of the drone swarm system, that is, the inherent "free movement" dynamic of the system.

[0067] According to the rigid body dynamics of a quadcopter (i.e., the above formula (3)):

[0068] Location dynamics: (The velocity term belongs to the drift state).

[0069] Speed ​​dynamics: (Gravity and disturbance belong to the drift state).

[0070] Posture dynamics: (Posture kinematics belongs to drift motion).

[0071] Angular velocity dynamics: (Gyroscopic torque and disturbance belong to the drift state).

[0072] It is these parts that are not directly driven by the control input, as shown in equation (8-1):

[0073] (8-1)

[0074] Describes the control input u i How does it affect the rate of change of state? This refers to the "gain" or "amplification factor" of the control input, which usually varies with the state (hence the term "state-dependent").

[0075] In formula (3), the control input only appears in:

[0076] Speed ​​dynamics: Thrust term .

[0077] Angular velocity dynamics: torque term .

[0078] Therefore, the input gain matrix As shown in equation (8-2):

[0079] (8-2)

[0080] S2. Based on the system model, define the value function for each quadcopter drone. Using the system model, value function, game definition, and Nash equilibrium conditions, construct a system of Hamilton-Jacobi partial differential equations. By solving the system of partial differential equations, establish the Nash feedback strategy for each quadcopter drone.

[0081] In one feasible implementation, each drone (or each controller / task unit) in the drone swarm is treated as a game player. Long-term performance metrics are defined for each player, simultaneously reflecting: penalties for deviations from the formation reference / target trajectory, coordination errors in maintaining the desired distance / angle with neighboring drones, control energy consumption (motor power, control surface / thrust variations, etc.), and penalties for risks related to safety constraints (such as minimum spacing, collision risk, and no-fly zone boundaries). This constructs a continuous-time non-zero-sum game, enabling each player to seek its own optimal and mutually consistent Nash equilibrium strategy under mutually coupled swarm dynamics.

[0082] Optionally, the specific operation of S2 may include the following S21-S25:

[0083] S21. Based on the sum of position tracking error, velocity consistency cost, formation maintenance cost, collision avoidance risk cost, and attitude and angle cost, construct the state cost of each quadcopter UAV. Based on the state cost, the control inputs of other quadcopter UAVs, and the control input weight matrix, construct the value function of each quadcopter UAV.

[0084] In one feasible implementation, each quadcopter drone in the cluster is regarded as a game player, and the control decision of player i is its thrust / torque input. The input set of other quadcopter UAVs is denoted as To reflect the non-zero-sum game characteristics of drone swarms, which require both cooperation to maintain formation and reduction of energy consumption to increase safety margin, the long-term value function of player i is defined as follows (9) based on the quadcopter drone swarm system model (7) in step 1:

[0085] (9)

[0086] in For the input weight matrix ( This describes "how the thrust / torque magnitude of the i-th UAV affects the overall cost of the i-th quadcopter UAV". State Cost Explicit cluster task representation: Let the desired position and velocity of the i-th UAV be respectively... and The relative deviation of the cluster formation is The neighbor set is (Determined by the communication / sensing topology), the minimum safe distance is , The specific form is as follows (10):

[0087] (10)

[0088] in These are positive semi-definite matrices, corresponding to the weights of position tracking, velocity consistency, formation maintenance, attitude, and angular velocity, respectively. Collision avoidance weights; collision avoidance potential function Take the differentiable barrier form to approximate the distance. The cost increases rapidly, as shown in equation (11):

[0089] (11)

[0090] Formula (11) indicates that when the distance between the two machines near When the denominator approaches 0, the thrust / torque strategy of the i-th UAV is forced to generate an evasive maneuver.

[0091] S22. According to game theory, the value function of minimizing each quadcopter drone is set as the individual optimization objective of each player in a non-zero-sum game. The non-zero-sum game is constructed as follows (12):

[0092] (12)

[0093] in, Let i represent the value function of the i-th quadcopter UAV. This represents the control input for all quadcopter UAVs except the i-th one. This represents the set of allowed control policies.

[0094] S23. Based on the individual optimization objectives of the quadcopter UAV, construct the Nash equilibrium conditions satisfied by the Nash strategy.

[0095] In one feasible implementation, formula (12) constitutes a non-zero-sum game of a quadcopter drone swarm, and its Nash strategy... The Nash equilibrium condition is satisfied, as shown in equation (13):

[0096] (13)

[0097] in, It represents a subset of the system's state space, usually referring to the range of values ​​for state X; Indicates the state region The set of permissible control policies.

[0098] S24. Based on the system model of the quadcopter drone swarm, the long-term cost function of each quadcopter drone, the individual optimization objective of each player, and the principle of dynamic programming, the coupled Hamilton-Jacobi equations are obtained.

[0099] In one feasible implementation, based on the system model of the quadcopter drone swarm, the long-term cost function of each quadcopter drone, the individual optimization objective of each player, and the principle of dynamic programming, a nonlinear, high-dimensional, and strongly coupled set of HJ partial differential equations is obtained, as shown in equation (14):

[0100] (14)

[0101] S25. By solving the Hamilton-Jacobi equations, establish the Nash feedback strategy for each quadcopter UAV.

[0102] In one feasible implementation, the Nash feedback strategy for each quadcopter UAV can be obtained by solving the above formula (14), as shown in the following formula (15):

[0103] (15)

[0104] in, , Let be the gradient of the value function of the i-th quadcopter UAV with respect to the cluster state.

[0105] S3. Construct a reinforcement learning evaluation-execution network for each quadcopter drone. The evaluation network is constructed based on the value function, and the execution network is constructed based on the Nash feedback policy.

[0106] In one feasible implementation, to avoid directly solving the coupled HJ equations (14) that expand rapidly with the 12N-dimensional state in step 2, this embodiment of the invention constructs an "evaluation-execution" network for each UAV:

[0107] 1) The "evaluation" network is used to approximate the long-term performance evaluation (value evaluation) of the UAV in the current game environment. The network input can include the local state, the relative state with neighboring UAVs, formation reference information, mission phase information, etc., and adjustable weight parameters can be set.

[0108] 2) The “Execution” network is used to generate the online control strategy of the UAV. Its output is a control command that can be directly issued (such as desired acceleration / attitude angular velocity / thrust distribution, etc.). It is combined with UAV dynamics and actuator constraints to perform reasonable mapping and constraint processing to ensure that the output can be implemented and executed in the real flight control loop.

[0109] Optionally, the specific operation of S3 may include the following S31-S33:

[0110] S31. Select a set of basis functions and parameterize the optimal value function corresponding to each quadcopter UAV.

[0111] In one feasible implementation, for the i-th quadcopter UAV, a set of basis functions is selected. Value functions can be Parameterization, specifically as shown in equation (16):

[0112] (16)

[0113] in, For ideal weights, For parameterized residuals.

[0114] S32. Approximate the parameterized optimal value function to obtain the estimated value function, which is then determined as the value function for evaluating the network.

[0115] In one feasible implementation, a reinforcement learning "evaluation" network is used to approximate the value function of the i-th drone. Specifically, it is shown in formula (17):

[0116] (17)

[0117] in, Let be the evaluation weights obtained through online learning by the "evaluation" network corresponding to the i-th quadcopter UAV. In the initial state, You can initialize the matrix; the following steps will be used to update it gradually.

[0118] S33. Based on the value function of the evaluation network, the constructed Nash feedback policy, and the weight matrix of the execution network, construct the execution policy of the execution network.

[0119] In one feasible implementation, to obtain a control output that can be directly applied to flight, formula (17) is substituted into formula (15), and the reinforcement learning "execution" network is designed to output the thrust / torque feedback law of the machine, as shown in the following formula (18):

[0120] (18)

[0121] in, The gradient of the basis function with respect to the cluster state is used to map the impact of changes in the state of the quadrotor UAV cluster on the value to the control channel. To "execute" network weights; The four components correspond to the total thrust and three-axis control torque that the i-th quadcopter UAV should apply, structurally ensuring the "output quantity" as the actual executable quantity, and through... This reflects the energy consumption penalty of thrust / torque.

[0122] S4. Based on the historical state of the quadcopter drone cluster and the current execution network weights and weight matrix, construct regression vectors and cost functions for each quadcopter drone. Based on the regression vectors and cost functions, construct the composite evaluation error of the evaluation network.

[0123] In one feasible implementation, during swarm flight, the state, control inputs, instantaneous costs (energy, formation error, safety costs, etc.), and communication / sensing information of each UAV and its neighborhood are collected in real time. The evaluation network estimates and execution network outputs at the current moment, combined with historical data, to construct a composite evaluation learning error signal containing both "current weight estimates" and "historical operational information." This fully leverages the effective incentive information of the UAV swarm within a limited timeframe, avoiding insufficient learning information or slowed convergence due to relying solely on instantaneous data.

[0124] During the online flight of a quadcopter drone swarm, the swarm can record historical moments. status (Obtained from positioning / communication sensors) and control signals To fully utilize historical information, this invention uses the current execution weight at the current time t. At historical points The strategy that each UAV should adopt at that time should be reconstructed (18), as shown in the following formula (19):

[0125] (19)

[0126] Substituting formula (19) into formula (7), we calculate the system state when the current strategy is used at the assumed historical time. We then multiply the system state by the gradient of the basis function to construct the regression vector, as shown in formula (20).

[0127] (20)

[0128] in As given in step S1(8).

[0129] Similarly, based on the definition of instantaneous cost in step S2 (9), the cost of the quadcopter drone swarm task can be reconstructed as follows (21):

[0130] (twenty one)

[0131] in In step S2 (10), the above formula is defined to include historical formation / collision avoidance / attitude and thrust / torque energy consumption information of each UAV. Based on formulas (20) and (21), the composite "evaluation" error of the i-th quadcopter UAV "evaluation" network can be constructed, as shown in the following formula (22):

[0132] (twenty two)

[0133] If the composite error If the value approaches 0, it indicates that the current "evaluation" network fits the swarm task cost of the quadcopter drone better.

[0134] S5. Based on the composite evaluation error, regression vector, and integral filter containing forgetting factor, construct the filtered error energy and the filtered regression energy, and then construct the normalized objective function of the evaluation network.

[0135] In one feasible implementation, a filtering mechanism with a "forgetting factor" is introduced based on the composite error signal and regression information to weight and aggregate historical data: newer swarm maneuver information has a higher weight, while older information gradually decays, thereby "compressing" long-term historical trajectories into a finite-dimensional filtered state quantity, reducing storage and computational burden. Furthermore, a finite-excitation metric method is established based on the statistical properties of the filtered information matrix, ensuring that even without continuous strong maneuvers or long-term rich stimuli, as long as sufficient representative maneuvers / interactions occur within a finite time window, parameter updates are effective and the learning process can proceed.

[0136] Optionally, the specific operation of S5 may include the following S51-S54:

[0137] S51. Define an integral filter for the input information according to the preset forgetting factor, as shown in the following formula (23):

[0138] (twenty three)

[0139] Where e represents the natural constant, Indicates the forgetting factor, ; t represents the current time, y represents the historical time variable, and y represents the input information.

[0140] In one feasible implementation, the unlimited growth of historical data during the online operation of a quadcopter drone swarm can lead to unsustainable storage and computation. This invention employs an exponential decay filter to "compress" the historical information "as it flies." This applies to any time-varying signal. The integral operator is defined as shown in equation (23) above, where the forgetting factor is used to indicate that the quadcopter drone swarm relies more on recent flight data and downplays long-term data. To facilitate real-time implementation in the drone swarm system, (23) is equivalent to a first-order filter, as shown in equation (24) below:

[0141] (twenty four)

[0142] That is, for the input signal The filter states are configured separately, so the required filter integral can be obtained in real time without storing the entire historical trajectory.

[0143] S52. Input the composite evaluation error into the integral filter and calculate the error energy after filtering.

[0144] In one feasible implementation, the filtered error energy is calculated based on the following equation (25-1):

[0145] (25-1)

[0146] in, For filtering error energy, it represents the "weighted cumulative intensity of historical mission error + energy consumption error".

[0147] S53. Input the regression vector into the integral filter, and calculate the regression energy after filtering based on the sum of the filtered regression vector and the regularization term.

[0148] In one feasible implementation, the filtered regression energy is calculated based on the following equation (25-2):

[0149] (25-2)

[0150] in, For filtering regression energy, it represents "the intensity of the incentive effect of historical data on learning"; This is a regularization term to avoid weak incentives. The denominator is too small.

[0151] S54. The ratio of the filtered error energy to the filtered regression energy is defined as the normalization objective function for evaluating the network.

[0152] In one feasible implementation, to measure the fitting effect of the "evaluation" network for the i-th quadcopter UAV, based on the composite error (22) of the "evaluation" network in step 4, the "evaluation" network is defined in the time interval. The total error is defined as the integral-weighted cumulative error, as shown in equation (25-3):

[0153] (25-3)

[0154] S6. Based on the filtered regression energy, regression vector, reconstruction cost function and integral filter, construct the information matrix and information vector for each quadcopter UAV. When the information matrix is ​​full rank, construct the time operator based on the information matrix, and construct the finite excitation condition based on the minimum eigenvalue of the information matrix of the historical regression vector within a finite time window.

[0155] In one feasible implementation, considering the varying degrees of excitation in UAV swarms at different stages (takeoff and assembly, formation adjustment, obstacle avoidance and crossing, mission relocation, terminal hold, etc.), this invention constructs a time operator to filter and sort historical data within a given time window, automatically selecting the most representative moments or data segments with the most abundant information. Specifically, using the eigenvalue distribution, trace, or condition number of the filtered information matrix as selection criteria, the "most excitation-rich" key nodes are selected from the candidate moments to form a higher-quality learning information matrix, thereby improving the learning efficiency and robustness of the UAV swarm under limited maneuverability conditions.

[0156] Optionally, the specific operation of S6 may include the following S61-S63:

[0157] S61. The current time and the product of the regression vector and its transpose are used as inputs to the integral filter for filtering. Based on the product of the reciprocal of the filtered regression energy and the filtered regression data, the information matrix of each quadcopter UAV is constructed.

[0158] S62. The current time and the product of the regression vector and the reconstruction cost function are used as inputs to the integral filter for filtering. Based on the product of the reciprocal of the filtered regression energy and the filtered cost data, the information vector of each quadcopter UAV is constructed.

[0159] In one feasible implementation, in order to break away from the dependence of traditional online learning on PE conditions, this embodiment of the invention transforms "whether the stimulus is sufficient" into an online evaluation of the minimum eigenvalue of the information matrix. Based on the filtering operator, the information matrix and information vector of the i-th UAV are defined as follows (26):

[0160] (26)

[0161] in, It is "the amount of identifiable information formed by the historical maneuvers and control behaviors of quadcopter drone swarms". The larger its minimum eigenvalue, the more balanced the incentive provided by historical data in all eigendirections.

[0162] S63. Determine if the information matrix of each quadcopter UAV is full rank. If the result is no, the finite excitation condition is not met, the time window of historical data needs to be expanded, and return to execute S4. If the result is yes, select the time operator as the most excitation moment according to the information matrix.

[0163] In one feasible implementation, to automatically select the "most exciting" historical segment, this embodiment of the invention defines a time operator, as follows:

[0164] (27)

[0165] Based on (27), the present invention further provides finite excitation conditions, as follows:

[0166] (28)

[0167] in, The sliding time window length indicates that only the most recent The optimal time for information content is selected from the second-by-second flight data to adapt to the switching of UAV mission phases and environmental changes. Condition (28) only needs to be true within a limited window, which is equivalent to "the UAV swarm has completed a sufficient number of actions (such as formation changes, collision avoidance and flight around, acceleration, deceleration and turning) in a certain time period", so that the parameter update can be guaranteed to be effective without the need for continuous artificial injection of exploration noise; and the selection by the time operator can ensure that the information used is not weaker than the current information:

[0168] (29)

[0169] This allows the "most useful maneuver segment in history" to be explicitly embedded into the learning law.

[0170] S7. Based on the information matrix, information vector, and time operator, and with the principle of minimizing the normalized objective function of the evaluation network, the weights of the evaluation network are updated by gradient descent. Based on the updated weights of the evaluation network, the filtered regression energy, the integral filter, the regression vector, and the gradient of the basis function, the weights of the execution network are updated.

[0171] In one feasible implementation, an online update law for the evaluation network weights is designed based on the information matrix of composite evaluation error, filtered signal, and time operator selection. This update process is then constrained by stability analysis to ensure stable closed-loop flight of the UAV swarm, smooth command execution, and compliance with actuator limitations during the learning period. Furthermore, an execution network weight update law is designed to allow each UAV's strategy to gradually evolve and converge to near a Nash equilibrium within an interactive game environment. During online execution, the algorithm simultaneously performs two tasks: continuously updating network weights and continuously outputting real-time control inputs directly usable in the flight control loop.

[0172] Optionally, the specific operation of S7 may include the following S71-S73:

[0173] S71. The objective function to be minimized is determined as the composite error energy function of the evaluation network to be minimized. The gradient of the evaluation objective function is expanded and simplified. A normalized information matrix and a normalized information vector are introduced to obtain the gradient of the evaluation network objective function. The gradient of the evaluation network objective function is substituted into the gradient descent formula, and a constant factor is absorbed into the learning rate of the evaluation network to obtain the intermediate update formula of the evaluation network. The information matrix and information vector of the most excited moment selected by the time operator are substituted into the intermediate update formula to obtain the updated weight matrix of the evaluation network.

[0174] In one feasible implementation, based on the normalized objective function (25), information matrix (26), and time operator (27) obtained in steps S5 and S6, this invention first adapts the weights of the reinforcement learning "evaluation" network for quadcopter drone swarms. Gradient descent updates are performed based on the principle of minimizing the normalized residual energy (25). The specific operations are as follows:

[0175] (1) The general form of gradient descent update is as follows (30-1):

[0176] (30-1)

[0177] (2) Expand gradient:

[0178] From formula (25), its gradient can be obtained as (30-2):

[0179] (30-2)

[0180] (3) Calculation gradient:

[0181] From formulas (22) and (25-1), we can obtain that

[0182]

[0183] Therefore, we can conclude that:

[0184] (30-3)

[0185] (4) Calculation gradient:

[0186] It is a filter based on the squared norm of the regression vector, while σ i It does not directly depend on ,therefore:

[0187] (30-4)

[0188] (5) Obtain the simplified gradient:

[0189]

[0190] (6) Substitution :

[0191]

[0192] so:

[0193]

[0194] (7) Substituting formula (26) into the equation, we get:

[0195]

[0196] (8) Incorporate a constant factor into the learning rate of the evaluation network:

[0197]

[0198] (9) Order ,but:

[0199] (30)

[0200] in, The reinforcement learning "evaluation" network learning rate determines the convergence speed of the drone swarm value function estimation; and It is obtained entirely from real flight data (position / velocity / attitude / angular velocity and thrust / torque) of quadcopter drone swarms through filtering and time screening.

[0201] S72. Determine the gradient based on the self-value function estimate by multiplying the filtered regression vector with the evaluation network weights. Determine the coupling effect of other players' policies on the self-policy update by filtering the product of the coupling term, the weight matrix of the execution network, the transpose of the weight matrix of the evaluation network, and the regression vector.

[0202] The coupling term is the product of the basis function gradient matrix, the input dynamics, the inverse of the self-control cost weight matrix, the cost coupling matrix of other players to themselves, the transpose of the input dynamics, and the transpose of the basis function gradient matrix.

[0203] In one feasible implementation, the gradient estimated based on the self-valued function is expressed as follows (32-1):

[0204] (31-1)

[0205] The coupling term is used to quantify the influence of other players' (drones') policies on the gradient of the native value function through the coupling of dynamics and cost, and its expression is as follows (31-2):

[0206] (31-2)

[0207] The expression for the coupling effect of other players' strategies on one's own strategy update is as follows (31-3):

[0208] (31-3)

[0209] S73. Based on the gradient estimated by the value function, the coupling effect of other players' policies on the update of the policy, the learning rate of the execution network, the gain of the execution network, the gain of the evaluation network, the updated weight matrix of the evaluation network, and the filtered regression energy, the updated weight matrix of the execution network is obtained.

[0210] In one feasible implementation, under the action of the "evaluation" network update law (30), the reinforcement learning "execution" network of the quadcopter drone swarm has adaptive weights. It can be designed as follows (32):

[0211] (32)

[0212] in, To enhance the "execution" of online learning, For gain,

[0213] Since the signals in the reinforcement learning "evaluation-execution" network update laws (30) and (32) can be obtained from the real-time / historical state and control variables of the quadcopter drone swarm in step 1, Calculations show that the aforementioned update law can achieve effective learning and parameter convergence using current and historical data without relying on PE conditions, thereby obtaining an approximate Nash strategy for the non-zero-sum game of quadcopter drone swarms. .

[0214] S8. Based on the updated weights of the evaluation network, the weights of the execution network, and the current state variables and control inputs of the quadcopter drone swarm, determine the approximate Nash feedback strategy and obtain the control commands for the quadcopter drone swarm.

[0215] In one feasible implementation, as the learning process progresses, the embodiments of the present invention enable the "evaluation-execution" network weights of each player in the UAV swarm to gradually converge to a stable solution, obtaining an approximate Nash strategy for the corresponding non-zero-sum game problem. The final output is the value assessment results of each UAV and an implementable feedback control strategy (which can serve as a command source for upper-layer collaborative control / middle-layer guidance / lower-layer attitude control), deployed in a real UAV swarm system to achieve complex tasks such as formation maintenance, formation reconfiguration, target tracking, cooperative collision avoidance, energy consumption optimization, and anti-jamming, while maintaining good engineering performance under conditions of limited computing power and communication.

[0216] In one feasible implementation, the convergent Nash feedback strategy is obtained by updating online according to steps S1-S7. Afterwards, the output is physically a "total thrust and three-axis torque command" that can be executed by the quadrotor, and can be directly used as the outer loop output of the flight control system.

[0217] The flight control interface provides angular velocity information for the four motors. (rad / s), then the thrust / torque can be mapped to the square of the motor speed through standard control allocation (mixed control): Let the arm length of the i-th UAV be... (m), lift coefficient is (N / (rad / s)2), the anti-torque coefficient is (N·m / (rad / s)2), then

[0218] (33)

[0219] matrix The first line indicates that the sum of the lift of the four motors of the i-th quadcopter drone equals the total thrust. The second and third lines indicate the thrust around the drone's body. The torque on the shaft is obtained by multiplying the difference in motor lift by the arm length. The fourth line indicates that the yaw torque of the quadcopter UAV is generated by the difference in drag torque caused by the motor rotation direction. From (33), we can obtain

[0220] (34)

[0221] This transforms the non-zero-sum game Nash equilibrium strategy obtained in steps S1-S7 into motor commands for the quadcopter drone swarm, enabling online closed-loop deployment of tasks such as formation keeping, cooperative tracking, and collision avoidance.

[0222] This invention proposes a non-zero-sum game-theoretic composite reinforcement learning method for cooperative control of UAV swarms. It utilizes the "evaluation-execution" network of each UAV to directly approximate value assessment and cooperative strategies online within the "state-control input" data space of the swarm flight. This avoids repeated offline solutions to the high-dimensional coupled HJ equations under strongly coupled nonlinear dynamics of UAVs, significantly reducing modeling accuracy requirements and computational overhead. By introducing a filtering mechanism with a "forgetting factor," weighted integration is performed on samples collected during the swarm's historical flight, including pose / velocity / relative formation errors, control commands, and energy consumption and safety costs. This constructs a composite "evaluation" network error and regression signal that integrates current and historical information. Furthermore, a finite incentive criterion that can be verified online is provided at the information matrix level, enabling the UAV swarm to maintain effective learning even when the continuous PE condition is not met. This avoids engineering risks such as formation jitter, increased energy consumption, and reduced collision avoidance margin caused by long-term injection of detection noise. Based on this, the "evaluation-execution" update law of composite reinforcement learning deeply integrates the current weight estimate with the filtered historical data, which can ensure that the network weights converge to the optimal neighborhood, thereby obtaining the value solution of the corresponding game problem, and enabling the control input of each UAV to converge to the Nash equilibrium strategy of the non-zero-sum game. While improving the accuracy of parameter identification and policy approximation, it also improves the formation maintenance, trajectory tracking and anti-disturbance performance of the cluster closed loop, and has good real-time performance, deployability and engineering application value.

[0223] Figure 3 This is a block diagram of a composite game-theoretic reinforcement learning device for quadrotor drone swarms provided in an embodiment of the present invention. This device is used for a composite game-theoretic reinforcement learning method for quadrotor drone swarms. (Refer to...) Figure 3 The device includes:

[0224] Unit 310 is used to establish a system model of the quadrotor drone swarm based on the state variables, control inputs, and rigid body dynamics model of each quadrotor drone.

[0225] Solver 320 is used to define the value function of each quadcopter UAV based on the system model. Through the system model, value function, game definition and Nash equilibrium conditions, Hamilton-Jacobi partial differential equation system is constructed. By solving the partial differential equation system, Nash feedback strategy of each quadcopter UAV is established.

[0226] The first building unit 330 is used to construct an evaluation-execution network for reinforcement learning for each quadcopter UAV, constructing an evaluation network based on a value function and an execution network based on a Nash feedback policy.

[0227] The second construction unit 340 is used to construct a regression vector and a cost function for each quadrotor drone based on the historical state of the quadrotor drone cluster and the current execution network weights and weight matrix, and to construct a composite evaluation error of the evaluation network based on the regression vector and cost function.

[0228] The third building unit 350 is used to construct the filtered error energy and the filtered regression energy based on the composite evaluation error, regression vector and integral filter containing forgetting factor, and then construct the normalized objective function of the evaluation network.

[0229] The fourth construction unit 360 is used to construct the information matrix and information vector of each quadcopter UAV based on the filtered regression energy, regression vector, reconstruction cost function and integral filter. When the information matrix is ​​full rank, a time operator is constructed based on the information matrix, and finite excitation conditions are constructed based on the minimum eigenvalue of the information matrix of the historical regression vector within a finite time window.

[0230] The update unit 370 is used to update the weights of the evaluation network by gradient descent based on the information matrix, information vector, and time operator, with the principle of minimizing the normalized objective function of the evaluation network. The update unit also updates the weights of the execution network based on the updated weights of the evaluation network, the filtered regression energy, the integral filter, the regression vector, and the gradient of the basis function.

[0231] The control unit 380 is used to determine an approximate Nash feedback strategy based on the updated weights of the evaluation network, the weights of the execution network, and the current state variables and control inputs of the quadcopter drone swarm, and to obtain control commands for the quadcopter drone swarm.

[0232] Figure 4 This is a schematic diagram of the structure of a composite game-theoretic reinforcement learning device for quadcopter drone swarms provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the composite game-theoretic reinforcement learning device for quadcopter drone swarms may include the above-mentioned... Figure 3 The illustrated composite game-themed reinforcement learning device for quadcopter drone swarms. Optionally, the composite game-themed reinforcement learning device 410 for quadcopter drone swarms may include a first processor 2001.

[0233] Optionally, the composite game-theoretic reinforcement learning device 410 for quadcopter drone swarms may also include a memory 2002 and a transceiver 2003.

[0234] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0235] The following is combined Figure 4 The components of the composite game-theoretic reinforcement learning device 410 for quadcopter drone swarms are described in detail below:

[0236] The first processor 2001 is the control center of the composite game-playing reinforcement learning device 410 for quadcopter drone swarms. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0237] Optionally, the first processor 2001 can perform various functions of the composite game reinforcement learning device 410 for quadcopter drone swarms by running or executing software programs stored in memory 2002 and calling data stored in memory 2002.

[0238] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0239] In a specific implementation, as one example, the composite game-theoretic reinforcement learning device 410 for quadcopter drone swarms may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0240] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0241] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the composite game reinforcement learning device 410 for quadcopter drone swarms (…). Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0242] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0243] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0244] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the composite game-playing reinforcement learning device 410 for quadcopter drone swarms. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0245] It should be noted that, Figure 4 The structure of the composite game reinforcement learning device 410 for quadcopter drone swarms shown in the figure does not constitute a limitation on the router. Actual composite game reinforcement learning devices for quadcopter drone swarms may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0246] Furthermore, the technical effect of the composite game reinforcement learning device 410 for quadcopter drone swarms can be referred to the technical effect of the composite game reinforcement learning method for quadcopter drone swarms described in the above method embodiments, and will not be repeated here.

[0247] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0248] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0249] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0250] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0251] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0252] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0253] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0254] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0255] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0256] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0257] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0258] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0259] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A composite game-theoretic reinforcement learning method for quadcopter drone swarms, characterized in that, The method includes: S1. Based on the state variables, control inputs, and rigid body dynamics model of each quadrotor drone swarm, establish a system model of the quadrotor drone swarm. S2. Based on the system model, define the value function of each quadcopter drone. Through the system model, value function, game definition, and Nash equilibrium conditions, construct the Hamilton-Jacobi partial differential equation system. By solving the partial differential equation system, establish the Nash feedback strategy for each quadcopter drone. S3. Construct a reinforcement learning evaluation-execution network for each quadcopter drone. Construct the evaluation network based on the value function and the execution network based on the Nash feedback policy. S4. Based on the historical state of the quadrotor drone cluster and the current execution network weights and weight matrix, construct regression vectors and reconstruction cost functions for each quadrotor drone. Based on the regression vectors and reconstruction cost functions, construct the composite evaluation error of the evaluation network. S5. Based on the composite evaluation error, regression vector and integral filter containing forgetting factor, construct the filtered error energy and the filtered regression energy, and then construct the normalized objective function of the evaluation network. S6. Based on the filtered regression energy, regression vector, reconstruction cost function and integral filter, construct the information matrix and information vector of each quadcopter UAV respectively. When the information matrix is ​​full rank, construct the time operator according to the information matrix, and construct the finite excitation condition according to the minimum eigenvalue of the information matrix of the historical regression vector within a finite time window. S7. Based on the information matrix, information vector, and time operator, and with the principle of minimizing the normalized objective function of the evaluation network, the weights of the evaluation network are updated by gradient descent. Based on the updated weights of the evaluation network, the filtered regression energy, the integral filter, the regression vector, and the gradient of the basis function, the weights of the execution network are updated. S8. Based on the updated weights of the evaluation network, the weights of the execution network, and the current state variables and control inputs of the quadcopter drone swarm, determine the approximate Nash feedback strategy and obtain the control commands for the quadcopter drone swarm. S6 includes: S61. Use the current time and the product of the regression vector and the transpose of the regression vector as input to the integral filter for filtering. Construct the information matrix of each quadcopter UAV based on the product of the reciprocal of the filtered regression energy and the filtered regression data. S62. The current time and the product of the regression vector and the reconstruction cost function are used as inputs to the integral filter for filtering. Based on the product of the reciprocal of the filtered regression energy and the filtered cost data, the information vector of each quadcopter UAV is constructed. S63. Determine whether the information matrix of each quadcopter drone is full rank. If the result is no, the finite excitation condition is not met, the time window of historical data needs to be expanded, and return to execute S4. If the result is yes, select the time operator as the most excitation moment according to the information matrix.

2. The composite game-theoretic reinforcement learning method for quadrotor drone swarms according to claim 1, characterized in that, S1 establishes a system model of the quadrotor drone swarm based on the state variables, control inputs, and rigid body dynamics model of each quadrotor drone, including: S11. In the inertial coordinate system, construct the state variables of each quadcopter UAV based on its center of mass position, center of mass velocity, roll, pitch and yaw Euler angles, and body angular velocity. S12. Based on the total thrust and three-axis control torque of each quadcopter drone, construct the control input for each quadcopter drone; S13. Construct a rigid body dynamics model for each quadrotor UAV based on its state variables, control inputs, translational disturbances, rotational disturbances, rotation matrix, and Euler angular rate mapping matrix. S14. Stack the state variables of all quadcopter drones to obtain the state variables of the quadcopter drone cluster. Stack the control inputs of all quadcopter drones to obtain the control inputs of the quadcopter drone cluster. S15. Based on the state variables, control inputs, and rigid body dynamics model of each quadrotor UAV swarm, the system model of the quadrotor UAV swarm is expressed in affine form as follows (1): （1） in, The state variables represent the state variables of a quadcopter drone swarm. Let N represent the control input for the i-th quadrotor UAV, where i represents the i-th quadrotor UAV and N represents the number of quadrotor UAVs. Indicates the drift state. This indicates that the input is dynamic.

3. The composite game-theoretic reinforcement learning method for quadrotor drone swarms according to claim 1, characterized in that, Based on the system model, S2 defines the value function for each quadcopter drone. Using the system model, value function, game theory definition, and Nash equilibrium conditions, it constructs a system of Hamiltonian-Jacobi partial differential equations. By solving this system of equations, it establishes the Nash feedback strategy for each quadcopter drone, including: S21. Based on the sum of position tracking error, velocity consistency cost, formation maintenance cost, collision avoidance risk cost, and attitude and angle cost, construct the state cost of each quadcopter UAV. Based on the state cost, the control input of other quadcopter UAVs, and the control input weight matrix, construct the value function of each quadcopter UAV. S22. According to game theory, the value function of minimizing each quadcopter drone is set as the individual optimization objective of each player in a non-zero-sum game. The non-zero-sum game is constructed as follows (2): （2） in, Let i represent the value function of the i-th quadcopter UAV. This represents the control input for all quadcopter UAVs except the i-th one. This represents the set of allowed control policies. This indicates the compact set in which the quadcopter drone swarm resides; S23. Based on the individual optimization objectives of the quadcopter UAV, construct the Nash equilibrium conditions satisfied by the Nash strategy; S24. Based on the system model of the quadcopter drone swarm, the long-term cost function of each quadcopter drone, the individual optimization objective of each player, and the principle of dynamic programming, the coupled Hamilton-Jacobi equations are obtained. S25. By solving the Hamilton-Jacobi equations, establish the Nash feedback strategy for each quadcopter UAV.

4. The composite game-theoretic reinforcement learning method for quadrotor drone swarms according to claim 1, characterized in that, The S3 constructs a reinforcement learning evaluation-execution network for each quadcopter UAV, building an evaluation network based on a value function and an execution network based on a Nash feedback policy, including: S31. Select a set of basis functions and parameterize the optimal value function corresponding to each quadcopter UAV; S32. Approximate the parameterized optimal value function to obtain the estimated value function, and determine it as the value function of the evaluation network; S33. Based on the value function of the evaluation network, the constructed Nash feedback policy, and the weight matrix of the execution network, construct the execution policy of the execution network.

5. The composite game-theoretic reinforcement learning method for quadrotor drone swarms according to claim 1, characterized in that, The S5 method, based on the composite evaluation error, regression vector, and integral filter containing a forgetting factor, constructs the filtered error energy and the filtered regression energy, and then constructs the normalized objective function of the evaluation network, including: S51. Define an integral filter for the input information according to the preset forgetting factor, as shown in equation (3): （3） Where e represents the natural constant, Indicates the forgetting factor, ; t represents the current time, y represents the historical time variable, and y represents the input information; S52. Input the composite evaluation error into the integral filter and calculate the error energy after filtering. S53. Input the regression vector into the integral filter, and calculate the filtered regression energy based on the sum of the filtered regression vector and the regularization term. S54. The ratio of the filtered error energy to the filtered regression energy is defined as the normalization objective function for evaluating the network.

6. The composite game-theoretic reinforcement learning method for quadrotor drone swarms according to claim 1, characterized in that, S7 updates the weights of the evaluation network using gradient descent based on the information matrix, information vector, and time operator, with the principle of minimizing the normalized objective function of the evaluation network. Based on the updated weights of the evaluation network, the filtered regression energy, the integral filter, the regression vector, and the gradient of the basis functions, the weights of the execution network are updated, including: S71. The objective function to be minimized is determined as the composite error energy function of the evaluation network to be minimized. The gradient of the evaluation objective function is expanded and simplified. A normalized information matrix and a normalized information vector are introduced to obtain the gradient of the evaluation network objective function. The gradient of the evaluation network objective function is substituted into the gradient descent formula, and a constant factor is absorbed into the learning rate of the evaluation network to obtain the intermediate update formula of the evaluation network. The information matrix and information vector of the most exciting moment selected by the time operator are substituted into the intermediate update formula to obtain the updated weight matrix of the evaluation network. S72. Determine the gradient based on the self-value function estimation based on the dot product of the filtered regression vector and the evaluation network weights; determine the coupling effect of other players' policies on the self-policy update based on the filtering result of the product of the coupling term, the weight matrix of the execution network, the transpose of the weight matrix of the evaluation network, and the regression vector; wherein, the coupling term is the product of the basis function gradient matrix, the input dynamics, the inverse of the self-control cost weight matrix, the cost coupling matrix of other players on itself, the transpose of the input dynamics, and the transpose of the basis function gradient matrix; S73. Based on the gradient estimated by the value function, the coupling effect of other players' policies on the update of the policy, the learning rate of the execution network, the gain of the execution network, the gain of the evaluation network, the updated weight matrix of the evaluation network, and the filtered regression energy, the updated weight matrix of the execution network is obtained.

7. A composite game-theoretic reinforcement learning device for quadrotor drone swarms, wherein the composite game-theoretic reinforcement learning device for quadrotor drone swarms is used to implement the composite game-theoretic reinforcement learning method for quadrotor drone swarms as described in any one of claims 1-6, characterized in that, The device includes: Establishment unit, used to establish system model of quadrotor drone swarm based on state variables, control inputs and rigid body dynamics model of each quadrotor drone; The solution unit is used to define the value function of each quadcopter UAV based on the system model. Through the system model, value function, game definition and Nash equilibrium conditions, the Hamilton-Jacobi partial differential equation system is constructed. By solving the partial differential equation system, the Nash feedback strategy of each quadcopter UAV is established. The first building unit is used to construct an evaluation-execution network for reinforcement learning for each quadcopter drone. The evaluation network is constructed based on the value function, and the execution network is constructed based on the Nash feedback policy. The second construction unit is used to construct regression vectors and reconstruction cost functions for each quadrotor drone based on the historical state of the quadrotor drone cluster and the current execution network weights and weight matrices. Based on the regression vectors and reconstruction cost functions, the composite evaluation error of the evaluation network is constructed. The third building unit is used to construct the filtered error energy and the filtered regression energy based on the composite evaluation error, regression vector and integral filter containing forgetting factor, and then construct the normalized objective function of the evaluation network. The fourth construction unit is used to construct the information matrix and information vector of each quadcopter UAV based on the filtered regression energy, regression vector, reconstruction cost function and integral filter. When the information matrix is ​​full rank, a time operator is constructed based on the information matrix, and finite excitation conditions are constructed based on the minimum eigenvalue of the information matrix of the historical regression vector within a finite time window. The update unit is used to update the weights of the evaluation network by gradient descent based on the information matrix, information vector, and time operator, with the principle of minimizing the normalized objective function of the evaluation network. The update unit updates the weights of the execution network based on the updated weights of the evaluation network, the filtered regression energy, the integral filter, the regression vector, and the gradient of the basis function. The control unit is used to determine an approximate Nash feedback strategy based on the updated weights of the evaluation network, the weights of the execution network, and the current state variables and control inputs of the quadcopter drone swarm, and to obtain control commands for the quadcopter drone swarm.

8. A composite game-theoretic reinforcement learning device for quadcopter drone swarms, characterized in that, The composite game-theoretic reinforcement learning device for quadcopter drone swarms includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images