Air-ground heterogeneous unmanned cluster traffic control and formation obstacle avoidance control method and system based on reinforcement learning

Abstract

The present disclosure provides a kind of air-ground heterogeneous unmanned cluster traffic control and formation obstacle avoidance control method and system based on reinforcement learning, comprising: obtaining the stage privacy protection mechanism of air-ground heterogeneous unmanned cluster according to the unmanned vehicle dynamics model and unmanned vehicle dynamics model;Using the composite guide vector field that fusion is attracted force field and repulsive force field, determine the integrated error of the air-ground heterogeneous unmanned cluster formation;Construct non-fragile performance constraint function, and obtain error conversion function according to the integrated error;And according to the error conversion function, determine the distributed reinforcement learning virtual controller and actual controller;According to the distributed reinforcement learning virtual controller and actual controller, realize the air-ground heterogeneous unmanned cluster prescribed time consistency formation control.The present disclosure can prevent malicious nodes from deriving the initial position and speed trajectory of other nodes through state interaction.

Description

Technical Field

[0001] This disclosure belongs to the field of aerospace technology, and in particular relates to a method and system for traffic control and formation obstacle avoidance control of air-to-ground heterogeneous unmanned swarms based on reinforcement learning. Background Technology

[0002] With the development of intelligent transportation systems and network communications, unmanned equipment, mainly including quadcopter drones and unmanned vehicles (UAVs), is widely used in patrol missions. However, a single patrol mode relying solely on the ground-based perspective of UAVs has limitations in terms of perception range and information acquisition. Fortunately, heterogeneous air-ground UAV / UAV systems can overcome these shortcomings. On the one hand, UAVs can perform patrol missions within the visual coverage of drones, thereby improving patrol efficiency. On the other hand, the payload capacity of UAVs can be used to provide continuous power support for quadcopter drones, thereby extending their patrol range. Therefore, researching distributed formation strategies for heterogeneous UAV and UAV systems is of great significance. In patrol missions, drones and UAVs often need to traverse complex areas, requiring obstacle avoidance and control capabilities.

[0003] To ensure the safe operation of the system in environments with obstacles, integrating obstacle avoidance strategies into the traffic management strategy of formation control algorithms is crucial. Artificial potential field methods, as a traditional path planning algorithm, are widely used in research on local obstacle avoidance control technology for multi-agent systems. However, the stability boundary proof of obstacle avoidance methods based on potential field methods relies on calculating the local forces exerted by obstacles on the moving target. Subsequently, by employing an iterative convergence strategy in the optimal algorithm, the problems existing in these obstacle avoidance methods are solved. However, the motion changes caused by obstacle avoidance may lead to trajectory deviations from the boundaries of physical safety constraints, thus introducing singularity problems.

[0004] When external disturbances or active obstacle avoidance deviations cause the tracking error to approach the predetermined envelope boundary, the control system may exhibit singularity issues, which are considered a vulnerability problem faced by current preset performance control methods. To eliminate the vulnerability of classical preset performance control methods, a multi-control strategy based on adjustable error constraint boundaries has been proposed to perceive the risk of error fluctuations by establishing a safety boundary, which serves as the trigger condition for dynamic reconfiguration. Therefore, establishing a safe corridor that allows for flexible deviations by employing a non-vulnerable preset performance control method is a worthy research problem. Furthermore, how to achieve optimal formation control of UAVs and unmanned vehicles based on preset performance control remains an important issue.

[0005] For platooning systems in traffic control strategies, the optimal cooperative control problem is often related to solving the Hamilton-Jacobi-Bellman (HJB) equations. However, due to the influence of uncertain system dynamics, it is usually difficult to solve the HJB equations analytically in practical applications. Introducing reinforcement learning methods to approximate the solution of the HJB equations is a promising approach, but its overall performance remains limited in applications requiring precise convergence time. This disclosure presents research on the optimal platooning control problem using reinforcement learning, aiming to ensure the convergence of heterogeneous UAV and unmanned vehicle systems within a specified time.

[0006] Furthermore, the security of heterogeneous UAV and autonomous vehicle systems in complex environments is crucial. This security includes communication security, defense against cyberattacks, and privacy protection. In recent years, significant progress has been made in privacy protection mechanisms based on masking functions. One study designed a privacy protection mechanism for the initial stage of a multi-agent system based on masking functions. To overcome the irreversible defects caused by the failure of existing masking functions, the challenge for future research lies in designing a staged privacy protection mechanism.

[0007] Therefore, this disclosure provides a method for traffic management and formation obstacle avoidance control of heterogeneous unmanned swarms in air and ground that can prevent malicious nodes from deducing the initial position and velocity trajectory of other nodes through state interaction. Summary of the Invention

[0008] The purpose of this disclosure is to provide a method and system for traffic management and formation obstacle avoidance control of heterogeneous unmanned swarms based on reinforcement learning, which is used to solve at least one technical problem in the prior art.

[0009] The technical solution disclosed herein is:

[0010] A reinforcement learning-based method for traffic management and formation obstacle avoidance control of heterogeneous air-to-ground unmanned swarms includes:

[0011] Establish dynamic models for both unmanned aerial vehicles (UAVs) and unmanned vehicles.

[0012] A phased privacy protection mechanism for air-to-ground heterogeneous unmanned clusters is obtained based on the aforementioned UAV dynamics model and unmanned vehicle dynamics model.

[0013] The integration error of the air-ground heterogeneous unmanned swarm formation is determined by using a composite guiding vector field that integrates attractive and repulsive fields.

[0014] Construct a non-fragile performance constraint function, and obtain an error transformation function based on the ensemble error; then determine the distributed reinforcement learning virtual controller and the actual controller based on the error transformation function.

[0015] Based on the distributed reinforcement learning virtual controller and the actual controller, the formation integration error converges to the neighborhood of zero, thereby realizing the time-consistent formation control of the air-ground heterogeneous unmanned cluster.

[0016] Based on the aforementioned UAV dynamics model and unmanned vehicle dynamics model, a phased privacy protection mechanism for air-to-ground heterogeneous unmanned swarms is obtained, including:

[0017] Based on the staged privacy protection function of the air-ground heterogeneous unmanned cluster, obtain the hyperbolic tangent mask coordinate transformation based on the leader and followers;

[0018] Based on the hyperbolic tangent mask coordinate transformation based on leaders and followers, a staged privacy protection mechanism based on leaders and followers is constructed to flexibly configure the protection window according to task requirements, preventing malicious nodes from deducing the initial position and velocity trajectory of other nodes through state interaction.

[0019] The expression for the hyperbolic tangent mask coordinate transformation based on leaders and followers is:

[0020] ;

[0021] ;

[0022] in, and Represents the phased privacy protection function for the i-th follower; and Represents the phased privacy protection function for the leader; and This represents the weighting factor of the hyperbolic tangent mask function used by the follower; and This represents the weighting factor of the hyperbolic tangent mask function used by the leader; and This indicates the status information of a follower under covert communication. and This indicates the leader's status information under covert communication. and This indicates the follower's status information before the phased privacy protection mechanism takes effect; and This indicates the leader's status information prior to the implementation of the phased privacy protection mechanism. Indicates the time of the control process.

[0023] The phased privacy protection mechanism based on the leader-follower structure is represented as follows:

[0024] ;

[0025] ;

[0026] in, This represents the uncertain variable of the i-th follower after masking; Let represent the unknown bounded perturbation variable of the i-th follower after masking; This represents the control gain of the i-th follower after masking. This represents the control input of the i-th follower after masking. This represents the nonlinear term of the i-th follower after masking. This represents the control signal of the i-th follower after masking. This represents the uncertain variables of the virtual leader after masking. This represents the unknown bounded perturbation variable of the virtual leader after masking. This indicates the control signals of the virtual leader after masking. The first derivative represents the state information of the follower under covert communication; The second derivative represents the state information of the follower under covert communication; This represents the first derivative of the leader's state information under covert communication.

[0027] The construction of the non-fragile performance constraint function and the acquisition of the error transformation function based on the integration error include:

[0028] Determine the non-fragile performance constraint function based on the preset performance function and the non-fragile adaptive boundary variables;

[0029] Based on the formation integration error and the non-fragile performance constraint function, an error transformation function is generated to allow for flexible deviations and obtain a safe corridor;

[0030] The non-fragile performance constraint function is expressed as follows:

[0031]

[0032] in, Indicates the first Position integration error of a quadcopter drone; Indicates the first Attitude integration error of a quadcopter drone; Indicates the first The integrated error of the position and heading angle of the unmanned vehicle; Indicates the first The lower boundary of the position integration error constraint for a quadcopter UAV; Indicates the first The upper boundary of the positional integrated error constraint for a quadcopter UAV; Indicates the first The lower boundary of the attitude integration error constraint for a quadcopter UAV; Indicates the first The upper boundary of the attitude integration error constraint for a quadcopter UAV; Indicates the first The lower boundary of the integrated error constraint between the position and heading angle of the autonomous vehicle; Indicates the first The upper boundary of the integrated error constraint between the position and heading angle of the unmanned vehicle.

[0033] The distributed reinforcement learning virtual controller is represented as:

[0034] ;

[0035] ;

[0036] ;

[0037] in, This represents the communication topology of the k-th quadcopter UAV in the position direction; This represents the communication topology of the k-th quadcopter UAV in the attitude direction; This represents the communication topology of the i-th unmanned vehicle in terms of position and heading angle. Indicates the parameters of the first distributed reinforcement learning controller; Indicates the parameters of the second distributed reinforcement learning controller; Indicates the parameters of the third distributed reinforcement learning controller; This represents the precondition of the transformed derivative of the position and orientation error of the k-th quadcopter UAV; This represents the precondition of the transformed derivative of the attitude direction error for the k-th quadcopter UAV; This represents the precondition of the error transformation derivative of the i-th unmanned vehicle in the position and heading angle directions; Let represent the error transformation function in the position direction of the k-th quadcopter UAV; Let represent the error transformation function in the attitude direction of the k-th quadcopter UAV; Let represent the error transformation function of the i-th unmanned vehicle in terms of position and heading angle; Let denote the derivative of the first formation variable of the composite guidance vector field of the r-th quadrotor UAV; This represents the neural network input signal of the position direction of the k-th quadcopter UAV; This represents the input signal of the neural network for the attitude orientation of the k-th quadcopter UAV. This represents the neural network input signal of the i-th autonomous vehicle in terms of position and heading angle. The neural network represents the first identifier in the position direction of the k-th quadcopter UAV. The neural network representing the first recognizer of the k-th quadcopter UAV in the attitude direction; The neural network representing the first identifier of the i-th autonomous vehicle in terms of position and heading angle; This represents the first adaptive law of the k-th quadcopter UAV in the position direction; This represents the first adaptive rate of the k-th quadrotor UAV in the attitude direction; This represents the first adaptive rate of the i-th unmanned vehicle in terms of position and heading angle. This represents the optimal weight estimate of the neural network for the first actor in the position direction of the k-th quadcopter UAV. This represents the optimal weight estimate of the first actor neural network for the k-th quadrotor UAV in the attitude direction; This represents the optimal weight estimate of the neural network for the first actor of the i-th autonomous vehicle in terms of position and heading angle. This represents the first input signal of the neural network in the position direction of the k-th quadcopter UAV; This represents the first input signal of the neural network for the k-th quadcopter UAV in the attitude direction; This represents the first input signal of the neural network for the i-th unmanned vehicle in terms of position and heading angle. Let the first Gaussian function of the k-th quadrotor UAV be oriented in the position direction; Let the first Gaussian function of the k-th quadrotor UAV be oriented in the position direction; Let represent the first Gaussian function of the i-th unmanned vehicle in terms of position and heading angle;

[0038] The actual controller for distributed reinforcement learning is represented as:

[0039] ;

[0040] ;

[0041] ;

[0042] in, This represents the control gain of the k-th quadcopter UAV in the position direction; This represents the control gain of the k-th quadcopter UAV in the attitude direction; This represents the control gain of the i-th unmanned vehicle in terms of position and heading angle. Let represent the error variable in the position direction of the k-th quadcopter UAV. Let $\mathbf{k}$ represent the error variable in the attitude direction of the $k$-th quadcopter UAV. Let represent the error variables of the i-th unmanned vehicle in terms of position and heading angle; The second-order neural network represents the positional orientation of the k-th quadcopter UAV. The second-order neural network represents the attitude direction of the k-th quadcopter UAV. The second-recognition neural network represents the position and heading angle of the i-th autonomous vehicle. Indicates the parameters of the fourth distributed reinforcement learning controller; This represents the second adaptive law of the k-th quadcopter UAV in the position direction; This represents the second adaptive rate of the k-th quadrotor UAV in the attitude direction; This represents the second adaptive rate of the i-th unmanned vehicle in terms of position and heading angle; This represents the optimal weight estimate of the second actor neural network for the k-th quadcopter UAV in the position direction; This represents the optimal weight estimate of the second actor neural network for the k-th quadcopter UAV in the attitude direction; This represents the optimal weight estimate of the second actor neural network for the i-th unmanned vehicle in terms of position and heading angle. This represents the second input signal of the neural network in the position direction of the k-th quadcopter UAV; This represents the second input signal of the neural network for the k-th quadcopter UAV in the attitude direction; This represents the second input signal of the neural network for the i-th autonomous vehicle in terms of position and heading angle. This represents the second Gaussian function of the k-th quadrotor UAV in the position direction; Denotes the second Gaussian function of the k-th quadrotor UAV in the attitude direction; Let represent the second Gaussian function of the i-th unmanned vehicle in terms of position and heading angle.

[0043] The implementation of time-consistent formation control for the air-to-ground heterogeneous unmanned cluster includes:

[0044] By analyzing using Lyapunov stability theory, we prove that the overall Lyapunov function expression is:

[0045] ;

[0046] in, Let represent the set of all stability coefficients of the first and second Lyapunov functions; Let represent the remaining terms of the first and second Lyapunov functions.

[0047] The expression for the composite guiding vector field is:

[0048]

[0049] in, Indicates the number of drones; Indicates the number of driverless cars; This indicates the total number of drones and unmanned vehicles; This represents the first design parameter of the attraction-guided vector field; Represents the attraction-guided vector field; This represents the first design parameter of the repulsive force guiding vector field; This represents a vector field guided by repulsive force.

[0050] A reinforcement learning-based air-to-ground heterogeneous unmanned swarm formation obstacle avoidance control system, based on the aforementioned reinforcement learning-based air-to-ground heterogeneous unmanned swarm traffic control and formation obstacle avoidance control method, includes: a virtual leader module, an external and internal collision module, a non-fragile preset performance function module, a performance index module, a module for the r-th UAV and the j-th unmanned vehicle, a composite guidance vector field module, an error transformation function module, an HJB equation module, a recognition neural network update rate module, an actor neural network update rate module, a critic neural network update rate module, a reinforcement learning neural network module, a model uncertainty and external disturbance module, a virtual controller module, and an actual controller module;

[0051] The virtual leader module interacts with the composite guidance vector field module to generate the desired position and speed information of the pilot drone, and transmits the obtained desired position and speed information of the pilot drone to the composite guidance vector field module.

[0052] The external and internal collision module interacts with the composite guidance vector field module to generate position information of static obstacles, position information of the r-th UAV and position information of the j-th unmanned vehicle, and transmits the obtained position information of static obstacles, position information of the r-th UAV and position information of the j-th unmanned vehicle to the composite guidance vector field module.

[0053] The nonfragile preset performance function module interacts with the error transformation function module and the performance index module to generate the upper and lower boundaries of the nonfragile performance constraints, and then transmits the obtained upper and lower boundaries of the nonfragile performance constraints to the error transformation function module and the performance index module.

[0054] The r-th UAV and j-th unmanned vehicle module interacts with the composite guidance vector field module to obtain model uncertainty and uncertainty input from the external disturbance module, disturbance information and actual control signal input from the actual controller module, generate position and velocity information of the r-th UAV and j-th unmanned vehicle, and transmit the obtained position and velocity information of the r-th UAV and j-th unmanned vehicle to the composite guidance vector field module.

[0055] The composite guidance vector field module interacts with the error conversion function module to obtain the position and velocity information of the r-th UAV and the j-th unmanned vehicle and the position information of the static obstacle, generate the composite guidance force of the r-th UAV and the j-th unmanned vehicle, and transmit the obtained composite guidance force of the r-th UAV and the j-th unmanned vehicle to the error conversion function module.

[0056] The error conversion function module interacts with the performance index module to obtain the upper and lower boundaries of the composite guiding force and non-fragile performance constraints of the r-th UAV and the j-th unmanned vehicle, generate the error signals of the r-th UAV and the j-th unmanned vehicle, and transmit the obtained error signals of the r-th UAV and the j-th unmanned vehicle to the performance index module.

[0057] The performance index module interacts with the HJB equation module to generate a first performance evaluation index from the error signals of the r-th UAV and j-th unmanned vehicle input by the non-fragile performance constraint upper and lower boundaries and the error transformation function module, and then transmits the obtained first performance evaluation index to the HJB equation module.

[0058] The HJB equation module interacts with the recognition neural network update rate module, the actor neural network update rate module, and the critic neural network update rate module to generate the first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation, and then transmits the obtained first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation to the recognition neural network update rate module, the actor neural network update rate module, and the critic neural network update rate module, respectively.

[0059] The identifier neural network update rate module interacts with the reinforcement learning neural network module to obtain the first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation, generate the first identifier neural network weight matrix and the second identifier neural network weight matrix, and transmit the obtained first identifier neural network weight matrix and the second identifier neural network weight matrix to the reinforcement learning neural network module.

[0060] The actor neural network update rate module interacts with the reinforcement learning neural network module to obtain the first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation, generate the first actor neural network weight matrix and the second actor neural network weight matrix, and pass the obtained first actor neural network weight matrix and the second actor neural network weight matrix to the reinforcement learning neural network module.

[0061] The critic neural network update rate module interacts with the reinforcement learning neural network module to obtain the first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation, generate the first critic neural network weight matrix and the second critic neural network weight matrix, and pass the obtained first critic neural network weight matrix and second critic neural network weight matrix to the reinforcement learning neural network module.

[0062] The reinforcement learning neural network module obtains the weight matrices of the first and second recognizer neural networks, the weight matrices of the first and second actor neural networks, the weight matrices of the first and second critic neural networks, the weight matrices of the first and second critic neural networks, generates approximate values ​​of unknown terms, and transmits the obtained approximate values ​​of unknown terms to the actual controller module.

[0063] The virtual controller module interacts with the reinforcement learning neural network module to obtain approximate values ​​of unknown terms, generate virtual control signals, error variables, and a second performance evaluation index, and then transmits the obtained virtual control signals, error variables, and second performance evaluation index to the HJB equation module.

[0064] The actual controller module interacts with the r-th UAV and j-th unmanned vehicle modules to obtain approximate values ​​of unknown terms, generate actual control signals, and transmit the obtained actual control signals to the r-th UAV and j-th unmanned vehicle modules.

[0065] The model uncertainty and external disturbance module generates uncertainty and disturbance information, and transmits the obtained uncertainty and disturbance information to the r-th UAV and j-th unmanned vehicle module.

[0066] The beneficial effects of this disclosure include at least the following:

[0067] The method described in this disclosure, through a phased privacy protection mechanism based on a leader-follower structure, enables air-to-ground heterogeneous unmanned swarm systems to flexibly configure protection windows according to mission requirements. This prevents malicious nodes from deducing the initial positions and velocity trajectories of other nodes through state interactions. Compared to existing technologies, the privacy protection mechanism ensures secure real-time data exchange when the air-to-ground heterogeneous unmanned swarm passes through narrow passages, enabling formation changes under complex obstacle avoidance conditions. Furthermore, compared to existing technologies, the composite guidance vector field can guide the formation to the desired position while avoiding multiple static obstacles. The introduction of adaptive boundaries allows the formation to deviate from obstacle avoidance for safety, preventing it from overshooting the navigable area, resulting in more accurate convergence time for the reinforcement learning controller within a specified timeframe. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of the process of the air-ground heterogeneous unmanned swarm traffic management and formation obstacle avoidance control method based on reinforcement learning of the present invention.

[0069] Figure 2 This is a schematic diagram of the structure of the air-ground heterogeneous unmanned swarm traffic management and formation obstacle avoidance control method based on reinforcement learning of the present invention.

[0070] Figure 3 This is a schematic diagram of the communication topology of the air-to-ground heterogeneous unmanned swarm formation disclosed in this publication.

[0071] Figure 4 This is a schematic diagram of the three-dimensional spatial trajectory of an air-ground heterogeneous unmanned swarm formation according to an embodiment of this disclosure.

[0072] Figure 5 This is a schematic diagram of the two-dimensional spatial trajectory of an air-ground heterogeneous unmanned swarm formation according to an embodiment of this disclosure.

[0073] Figure 6 This is a schematic diagram illustrating the privacy protection effect of a heterogeneous air-to-ground unmanned swarm formation passing through a narrow passage according to an embodiment of this disclosure.

[0074] Figure 7 This is a schematic diagram illustrating the position and attitude angle error constraints of a quadcopter UAV according to an embodiment of this disclosure.

[0075] Figure 8 This is a schematic diagram illustrating the position and yaw angle error constraints of an unmanned vehicle according to an embodiment of this disclosure.

[0076] Figure 9 This is a schematic diagram of the actor neural network weights for an air-to-ground heterogeneous unmanned swarm formation according to an embodiment of this disclosure.

[0077] Figure 10 This is a schematic diagram of the critic neural network weights for an air-to-ground heterogeneous unmanned swarm formation according to an embodiment of this disclosure.

[0078] Figure 11This is a schematic diagram of the neural network weights for the identifier of an air-to-ground heterogeneous unmanned swarm formation according to an embodiment of this disclosure.

[0079] Figure 12 This is a schematic diagram of the reinforcement learning cost function for an air-to-ground heterogeneous unmanned swarm formation according to an embodiment of this disclosure. Detailed Implementation

[0080] The following describes specific embodiments of the present disclosure to enable those skilled in the art to understand the disclosure. However, it should be understood that the disclosure is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the disclosure as defined and determined by the appended claims. All disclosed creations utilizing the concept of the present disclosure are protected.

[0081] Note that the drone in this embodiment is a quadcopter drone. Specific Implementation Example 1: This disclosure provides an embodiment:

[0083] like Figure 1 As shown, a method for traffic management and formation obstacle avoidance control of heterogeneous unmanned swarms based on reinforcement learning includes the following steps:

[0084] S1. Establish dynamic models of a six-degree-of-freedom quadcopter UAV and a three-degree-of-freedom unmanned vehicle;

[0085] The expression for the dynamic model of the k-th six-DOF quadrotor UAV is as follows:

[0086] (1)

[0087] The expression for the dynamic model of the i-th three-degree-of-freedom autonomous vehicle is:

[0088] (2)

[0089] S2. Based on the dynamic models of a six-degree-of-freedom quadcopter UAV and a three-degree-of-freedom unmanned vehicle, determine the phased privacy protection mechanism for air-ground heterogeneous unmanned swarms.

[0090] S21. Based on the staged privacy protection function of the air-ground heterogeneous unmanned cluster, determine the hyperbolic tangent mask coordinate transformation based on the leader and followers;

[0091] Specifically, a phased privacy protection function is established for air-to-ground heterogeneous unmanned clusters:

[0092] (3)

[0093] (4)

[0094] In the formula, and This represents the privacy protection weighting coefficient for followers. and This represents the leader's privacy protection weighting coefficient. Indicates the sequence number that triggered the event. and Indicates the start and end times of the phased privacy protection mechanism call. Indicates the time of the control process.

[0095] The expression for the hyperbolic tangent mask coordinate transformation based on leaders and followers is determined as follows:

[0096] (5)

[0097] (6)

[0098] S22. Based on the hyperbolic tangent mask coordinate transformation based on leaders and followers obtained in S21, construct a staged privacy protection mechanism based on leaders and followers;

[0099] The expression for constructing a phased privacy protection mechanism based on a leader-follower structure is as follows:

[0100] (7)

[0101] (8)

[0102] S3. Construct a composite guiding vector field that integrates the attraction and repulsion fields to determine the integration error of the air-ground heterogeneous unmanned swarm formation;

[0103] S31. Based on the design of the attractive potential energy field and the repulsive potential energy field, as well as the risk level of the surrounding area, determine the composite guiding vector field;

[0104] The expressions for the attractive potential energy field and the repulsive potential energy field are:

[0105] (9)

[0106] (10)

[0107] in, This represents the expected distance between the r-th follower and the j-th follower along the x-axis. This represents the expected distance between the r-th follower and the j-th follower along the y-axis. This represents the expected distance between the r-th follower and the virtual leader along the x-axis. This represents the expected distance between the r-th follower and the virtual leader along the y-axis. Indicates the design parameters of the first potential energy field. This represents the design parameters for the second potential energy field. This represents the design parameters for the third potential energy field. Indicates the design parameters of the attractive potential energy field. This represents the design parameters for the repulsive potential energy field.

[0108] Risk level of surrounding areas and The expression is:

[0109] (11)

[0110] (12)

[0111] In the formula, This indicates the risk coefficient for the first region. This indicates the risk coefficient for the second region. This represents the relative distance between the r-th follower and the j-th follower. This represents the relative distance between the r-th follower and the virtual leader. Indicates the radius of each obstacle. This represents the minimum safe distance that an unmanned swarm must maintain. Indicates the activation distance of obstacle avoidance behavior. and This represents the safety buffer radius set around the r-th quadcopter drone and the j-th unmanned vehicle. This represents the expected distance between the r-th follower and the j-th follower. This represents the expected distance between the r-th follower and the virtual leader.

[0112] Combining the attractive and repulsive potential energy fields with the risk level of the surrounding area, the expression for the composite guiding vector field is determined as follows:

[0113] (13)

[0114] S32. Determine the formation integration error of the air-to-ground heterogeneous unmanned swarm based on the formation position, speed, disturbance error and composite guidance vector field;

[0115] The expression for the formation position error between the r-th quadcopter UAV and the j-th unmanned vehicle is:

[0116] (14)

[0117] In the formula, This indicates the position and status information of the r-th quadcopter UAV. This represents the relative position and state information of the r-th quadcopter UAV. This represents the location and status information of the j-th driverless vehicle. This represents the relative position and state information of the j-th autonomous vehicle. This indicates the location and status information of the virtual leader. All are elements in a directed graph. ,and This indicates the communication relationship between the r-th quadcopter drone and the j-th unmanned vehicle. This indicates the communication relationship between the r-th quadcopter drone and the virtual leader.

[0118] The expression for the formation speed error between the r-th quadcopter UAV and the j-th unmanned vehicle is:

[0119] (15)

[0120] In the formula, This indicates the speed status information of the r-th quadcopter UAV. This represents the relative velocity state information of the r-th quadcopter UAV. This represents the speed and status information of the j-th autonomous vehicle. This represents the relative speed state information of the j-th autonomous vehicle. This indicates the speed status information of the virtual leader.

[0121] The expression for the formation disturbance error of the r-th quadcopter UAV and the j-th unmanned vehicle is:

[0122] (16)

[0123] In the formula, This represents the disturbance state information of the r-th quadcopter UAV. This represents the disturbance state information of the j-th autonomous vehicle.

[0124] The expression for the formation variables of the composite guidance vector field of the r-th quadrotor UAV and the j-th unmanned vehicle is:

[0125] (17)

[0126] In the formula, Denotes the coefficients of the first composite guiding vector field formation variables. This represents the coefficients of the formation variables of the second composite guiding vector field. This represents the composite guidance force of the r-th quadcopter UAV. Let represent the composite guiding force of the j-th unmanned vehicle.

[0127] The expression for the integration error of heterogeneous air-to-ground unmanned swarm formation is:

[0128] (18)

[0129] S4. Construct a non-fragile performance constraint function and generate an error transformation function based on the integration error of the air-ground heterogeneous unmanned swarm formation;

[0130] S41. Determine the non-fragile performance constraint function based on the preset performance function and the non-fragile adaptive boundary variables;

[0131] The expression for the default performance function is:

[0132] (19)

[0133] in, Indicates the first Initial values ​​of the performance function of each follower Indicates the first The final value of the performance function of each follower. Indicates the time of the control process. Indicates the convergence time.

[0134] The expression for the nonfragile adaptive boundary variable is:

[0135] (20)

[0136] In the formula, Indicates the first The location of the quadcopter drone, Indicates the first The attitude of the quadcopter drone, Indicates the first The position and heading angle of the autonomous vehicle This represents the set of state information for drones and unmanned vehicles. Indicates the lower bound. Indicates the upper bound. Denotes the boundary set, Represents the parameters of the first adaptive performance boundary variables. Represents the parameters for minimizing the adaptive performance boundary. This represents the parameter for maximizing the adaptive performance boundary. This represents the boundary optimal parameters for adaptive performance. Represents the adaptive performance boundary balance parameter. This represents the parameters of the second adaptive performance boundary variables. This represents the third adaptive performance boundary variable parameter. This represents the fourth adaptive performance boundary variable parameter. This represents the fifth adaptive performance boundary variable parameter.

[0137] The expression for the non-fragile performance constraint function is:

[0138] (twenty one)

[0139] S42. Generate an error transformation function based on the formation integration error obtained in S3 and the non-fragile performance constraint function obtained in S41;

[0140] The expression for the error transformation function is:

[0141] (twenty two)

[0142] S5. Determine the distributed reinforcement learning virtual controller and the actual controller based on the error transformation function;

[0143] Error transformation function Perform time differentiation:

[0144] (twenty three)

[0145] In the formula, and These represent the pre- and post-terms of the error transformation function, respectively. Further observation reveals... . Indicates communication link relationships, This represents the first disturbance state information of the leader. This step proposes a two-step iterative control method for designing the optimal virtual controller and the actual controller; by combining the first cost function and applying the principle of optimality, the first Hamilton-Jacobi-Bellman equation is proposed.

[0146] The first cost function is expressed as: ;

[0147] The first Hamilton-Jacobi-Bellman equation can be expressed as:

[0148] (twenty four)

[0149] In the formula, Represents the first cost function. Indicates virtual control rate. Indicates the first set of permissions. This can be considered as the optimal virtual control law. , This represents an approximation of the error transformation function. This represents the error constraint for air-to-ground heterogeneous unmanned swarm formation.

[0150] To estimate the uncertainty, a radial basis function-based recognizer neural network and an update rule are introduced:

[0151] (25)

[0152] In the formula, This represents the input to the neural network. This represents the weight matrix of the first recognizer's neural network. This represents the radial basis function vector of the first identifier's neural network. This represents the positive design parameters of the neural network of the first recognizer. This represents the positive design parameter indicating the first update rate.

[0153] Identify the actor neural network and the critic neural network and their update rules:

[0154] (26)

[0155] In the formula, This represents the input to the first actor neural network and the first critic neural network. Denotes the radial basis function vectors of the first actor neural network and the first critic neural network. This represents the first positive design parameter of the first actor neural network. This represents the second positive design parameter of the first actor neural network. This represents the first positive design parameter of the first critic neural network.

[0156] According to the optimality theorem, the equation is... Introducing neural networks into the seer-actor-critic framework:

[0157] (27)

[0158] In the formula, This represents the first Hamilton-Jacobi-Bellman equation. This represents the estimated value of the first cost function. This represents the input signal to the neural network of the recognizer. This represents the neural network of the first recognizer. This represents the first adaptive rate. This represents the optimal weight estimate of the first critic neural network. This represents the optimal virtual controller.

[0159] Based on the identifier-actor-critic neural network framework, the optimal virtual controller can be obtained:

[0160] (28)

[0161] For error variables Taking the time derivative, we get:

[0162] (29)

[0163] In the formula, Indicates control gain. Indicates the actual controller, Indicating model uncertainty, This indicates the second perturbation state information of the follower. Indicates the leader's control torque. This indicates the second perturbation state information of the leader. Let represent the derivative of the speed error of the j-th unmanned vehicle formation.

[0164] The second Hamilton-Jacobi-Bellman equation is obtained based on formula (29) and the second cost function;

[0165] The second cost function is: ;

[0166] The specific expression for the second Hamilton-Jacobi-Bellman equation is:

[0167] (30)

[0168] In the formula, This represents the second cost function. Indicates the second set of permissions. This represents the balance parameters of the actual controller.

[0169] Uncertainty is estimated by constructing a neural network for the identifyr and its update rules:

[0170] (31)

[0171] In the formula, This represents the weight matrix of the second-recognizer's neural network. This represents the radial basis function vector of the second identifier's neural network. This represents the positive design parameters of the second-recognition neural network. This represents the positive design parameter for the second update rate.

[0172] Identify the actor neural network and the critic neural network and their update rules:

[0173] (32)

[0174] In the formula, This represents the input to the second actor neural network and the second critic neural network. Denotes the radial basis function vectors of the second actor neural network and the second critic neural network. This represents the first positive design parameter of the second actor neural network. This represents the second positive design parameter of the second actor neural network. This represents the first positive design parameter of the second critic neural network.

[0175] Since the optimality theorem equation is: ;

[0176] According to the optimality theorem, the equation is:

[0177] (33)

[0178] In the formula, This represents the second Hamilton-Jacobi-Bellman equation. This represents the optimal actual controller. This represents the estimated value of the second cost function. This represents the neural network of the second recognizer. This represents the second adaptive rate. This represents the optimal weight estimate of the second critic's neural network.

[0179] Based on the above estimates, the optimal actual controller can be derived:

[0180] (34)

[0181] S6. Based on the distributed reinforcement learning virtual controller and the actual controller, the formation integration error is made to converge to the neighborhood of zero, thereby realizing the time-consistent formation control of the air-ground heterogeneous unmanned swarm.

[0182] To achieve time-consistent formation control of heterogeneous unmanned swarms, sufficient conditions are presented, including Lemma 1, Lemma 2, and Theorem 1.

[0183] Lemma 1: If any real vector and real numbers ,when and When both conditions are met, the following conclusion must hold:

[0184] (35)

[0185] Lemma 2: For the function and ,in , , and All are bounded functions. and It is continuously differentiable, when Hypothetical variables In all Within a certain range, it remains within the specified performance limits; the upper bound of the performance function. and the lower realm Meet the conditions This condition also applies. and .

[0186] Theorem 1: For the above uncertain nonlinear system, if the above lemma holds, then by employing actor neural networks, critic neural networks, recognizer neural networks and update rules, the error signals of the optimal virtual controller and the actual controller are bounded, consistent and eventually bounded.

[0187] Proof: Define the first Lyapunov function as:

[0188] (36)

[0189] In the formula, This represents the approximate error of the first adaptive rate. This represents the approximation error of the first actor neural network. This represents the approximation error of the first critic's neural network. This represents the approximation error of the first actor neural network.

[0190] Based on the recognizer neural network represented by formulas (18), (23), and (25), and the actor-critic neural network represented by formula (26), substituting the optimal virtual controller represented by formula (28) into formula (36) and taking its time derivative, we can obtain:

[0191] (37)

[0192] In the formula, This represents the first ideal weight matrix. This represents the first approximation error. Let represent the estimated value of the first adaptive rate. By rearranging formula (37) and combining it with Lemma 1, we can obtain:

[0193] (38)

[0194] In the formula, Denotes the first stability coefficient of the first Lyapunov function. This represents the second stability coefficient of the first Lyapunov function. This represents the third stability coefficient of the first Lyapunov function. This represents the fourth stability coefficient of the first Lyapunov function. This represents the fifth stability coefficient of the first Lyapunov function. Let represent the remaining terms of the first Lyapunov function.

[0195] Define the second Lyapunov function as:

[0196] (39)

[0197] In the formula, This represents the approximate error of the second adaptive rate. This represents the approximation error of the second-order recognizer's neural network. This represents the approximation error of the second critic's neural network. This represents the approximation error of the second actor neural network.

[0198] Similar to the steps above, combining Lemma 1 and the optimal practical controller represented by Equation (34), we can obtain:

[0199] (40)

[0200] In the formula, This represents the first stability coefficient of the second Lyapunov function. This represents the second stability coefficient of the second Lyapunov function. This represents the third stability coefficient of the second Lyapunov function. This represents the fourth stability coefficient of the second Lyapunov function. This represents the fifth stability coefficient of the second Lyapunov function. Let represent the remaining terms of the second Lyapunov function.

[0201] From the above discussion, the overall Lyapunov function can be derived:

[0202] (41)

[0203] A simple analysis of formula (41) shows that, ,in This represents the initial value of the overall Lyapunov function. Therefore, if the integration error... Then the condition The establishment of the system allows for further deduction of the specified time. Existence makes ,when Then, according to formulas (36) and (39), the conditions are... and Established, among which and For the inequality parameters. In summary, we can obtain... and The inequality indicates the error signal. and It satisfies the boundedness condition.

[0204] According to the error signal and The boundedness of can be seen from formula (22) for ,variable Strictly constrained by formula (21). According to Lemma 2, the error signal and All are semi-global eventually consistent bounded and within a specified time. It converges internally and always remains within the range defined by formula (21). Specific Implementation Example 2:

[0206] This disclosure also provides an embodiment:

[0207] In the embodiments disclosed herein, such as Figure 2 As shown, a control system can be constructed that includes a virtual leader module, an external and internal collision module, a non-fragile preset performance function module, a performance index module, a module for the r-th UAV and the j-th unmanned vehicle, a composite guidance vector field module, an error transformation function module, an HJB equation module, a recognition neural network update rate module, an actor neural network update rate module, a critic neural network update rate module, a reinforcement learning neural network module, a model uncertainty and external disturbance module, a virtual controller module, and an actual controller module. The virtual leader module is used to generate the desired position and desired velocity information of the leading UAV and transmit the obtained desired position and desired velocity information of the leading UAV to the composite guidance vector field module.

[0208] The external and internal collision module is used to generate the position information of static obstacles, the position information of the r-th UAV and the position information of the j-th unmanned vehicle, and transmit the obtained position information of static obstacles, the position information of the r-th UAV and the position information of the j-th unmanned vehicle to the composite guidance vector field module.

[0209] The nonfragile preset performance function module is used to generate the upper and lower boundaries of nonfragile performance constraints, and then pass the obtained upper and lower boundaries of nonfragile performance constraints to the error transformation function module and the performance index module.

[0210] The performance index module is used to obtain the upper and lower boundaries of the nonfragile performance constraints input by the nonfragile preset performance function module and the error signals of the r-th UAV and j-th unmanned vehicle input by the error transformation function module, further generate the first performance evaluation index, and pass the obtained first performance evaluation index to the HJB equation module.

[0211] The module for the r-th UAV and j-th unmanned vehicle is used to acquire the uncertainty and disturbance information input by the model uncertainty and external disturbance module and the actual control signal input by the actual controller module. It further generates the position and velocity information of the r-th UAV and j-th unmanned vehicle and transmits the obtained position and velocity information of the r-th UAV and j-th unmanned vehicle to the composite guidance vector field module.

[0212] The composite guidance vector field module is used to acquire the position and velocity information of the r-th UAV and j-th unmanned vehicle input from the r-th UAV and j-th unmanned vehicle module, and the position information of the static obstacle, the r-th UAV, and the j-th unmanned vehicle input from the external and internal collision module. It further generates the composite guidance force of the r-th UAV and j-th unmanned vehicle, and transmits the obtained composite guidance force of the r-th UAV and j-th unmanned vehicle to the error transformation function module.

[0213] The error transformation function module is used to obtain the composite guidance force of the r-th UAV and the j-th unmanned vehicle input by the composite guidance vector field module and the upper and lower boundaries of the non-fragile performance constraints input by the non-fragile preset performance function module. It further generates the error signals of the r-th UAV and the j-th unmanned vehicle and transmits the obtained error signals of the r-th UAV and the j-th unmanned vehicle to the performance index module.

[0214] The HJB equation module is used to obtain the first performance evaluation index input from the performance index module and the virtual control signal, error variable, and second performance evaluation index input from the virtual controller module. It further generates the first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation, and then passes the obtained first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation to the recognizer neural network update rate module, the actor neural network update rate module, and the critic neural network update rate module, respectively.

[0215] The Recognizer Neural Network Update Rate Module is used to obtain the first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation from the input of the HJB equation module, further generate the first Recognizer Neural Network Weight Matrix and the second Recognizer Neural Network Weight Matrix, and pass the obtained first Recognizer Neural Network Weight Matrix and the second Recognizer Neural Network Weight Matrix to the Reinforcement Learning Neural Network Module.

[0216] The actor neural network update rate module is used to obtain the first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation from the input of the HJB equation module, further generate the first actor neural network weight matrix and the second actor neural network weight matrix, and pass the obtained first actor neural network weight matrix and second actor neural network weight matrix to the reinforcement learning neural network module.

[0217] The critic neural network update rate module is used to obtain the first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation from the input of the HJB equation module, further generate the first critic neural network weight matrix and the second critic neural network weight matrix, and pass the obtained first critic neural network weight matrix and second critic neural network weight matrix to the reinforcement learning neural network module.

[0218] The reinforcement learning neural network module is used to obtain the first and second weight matrices of the recognizer neural network, the first and second weight matrices of the actor neural network, the first and second weight matrices of the critic neural network, which are input to the recognizer neural network update rate module, and the first and second weight matrices of the critic neural network update rate module. It further generates approximate values ​​of unknown terms and passes the obtained approximate values ​​of unknown terms to the actual controller module.

[0219] The model uncertainty and external disturbance module is used to generate uncertainty and disturbance information, and then transmits the obtained uncertainty and disturbance information to the r-th UAV and j-th unmanned vehicle module.

[0220] The virtual controller module is used to obtain approximate values ​​of unknown terms from the input of the reinforcement learning neural network module, further generate virtual control signals, error variables, and a second performance evaluation index, and then transmit the obtained virtual control signals, error variables, and second performance evaluation index to the HJB equation module.

[0221] The actual controller module is used to obtain approximate values ​​of unknown terms from the input of the reinforcement learning neural network module, further generate actual control signals, and transmit the obtained actual control signals to the r-th UAV and the j-th unmanned vehicle module.

[0222] Considering an air-to-ground heterogeneous swarm system consisting of one virtual lead drone, two follower drones, and four follower unmanned vehicles, a simulation experiment was conducted using the dynamic model represented by formulas (1) and (2) to verify the effectiveness of the designed reinforcement learning-based air-to-ground heterogeneous unmanned swarm traffic control and formation obstacle avoidance control method. The model parameters of the air-to-ground heterogeneous unmanned swarm system are shown in Table 1, the initial parameters of the simulation experiment are shown in Table 2, and the parameters of the control method are shown in Table 3. The communication topology between the various unmanned systems is described based on an adjacency network. Figure 3 In the diagram, 0 represents the virtual leader, 1-2 represent two follower drones, and 3-6 represent four follower unmanned vehicles.

[0223] Table 1: Model Parameters of Heterogeneous Unmanned Cluster System

[0224]

[0225] In Table 1, This indicates the mass of the first quadcopter drone. This indicates the mass of the second quadcopter drone. This indicates the mass of the first driverless car. This indicates the mass of the second driverless car. This indicates the mass of the third driverless car. This indicates the mass of the fourth driverless car. This indicates that the k-th quadcopter drone is in Impedance factor in the axial direction, This indicates that the k-th quadcopter drone is in Impedance factor in the axial direction, This indicates that the k-th quadcopter drone is in Impedance factor in the axial direction, This represents the impedance factor of the k-th quadcopter UAV in the roll direction. This represents the impedance factor of the k-th quadcopter UAV in the pitch direction. This represents the impedance factor of the k-th quadcopter UAV in the yaw direction. This indicates that the k-th quadcopter drone is in Inertial torque in the axial direction, This indicates that the k-th quadcopter drone is in Inertial torque in the axial direction, This indicates that the k-th quadcopter drone is in Inertial torque in the axial direction, This represents the moment of inertia of the rotor blades of the k-th quadcopter UAV. Indicates that the i-th driverless car is in Damping matrix in the axial direction, Indicates that the i-th driverless car is in Damping matrix in the axial direction, This indicates that the k-th quadcopter drone is in External disturbances in the axial direction, This indicates that the k-th quadcopter drone is in External disturbances in the axial direction, This indicates that the k-th quadcopter drone is in External disturbances in the axial direction, This represents the external interference affecting the k-th quadcopter UAV in the roll angle direction. This represents the external interference affecting the k-th quadcopter UAV in the pitch direction. This represents the external interference affecting the k-th quadcopter UAV in the yaw direction. Indicates that the i-th driverless car is in External disturbances in the axial direction, Indicates that the i-th driverless car is in External disturbances in the axial direction, It represents the acceleration due to gravity.

[0226] Table 2: Initial parameters of the simulation experiment

[0227]

[0228] Table 3: Control Method Parameters

[0229]

[0230] In Table 3, This represents the first privacy protection weight coefficient of the i-th follower. This represents the second privacy protection weight coefficient for the i-th follower. This represents the weighting factor of the first hyperbolic tangent mask function used by the i-th follower. This represents the weighting factor of the second hyperbolic tangent mask function used by the i-th follower. Indicates the first The start time of each phased privacy protection mechanism call. Indicates the first The end time of each phased privacy protection mechanism call. This represents the primary privacy protection weighting coefficient for virtual leaders. The second privacy protection weighting coefficient represents the virtual leader. This represents the weighting factor of the first hyperbolic tangent mask function used by the virtual leader. This represents the weighting factor of the second hyperbolic tangent mask function used by the virtual leader. Indicates the design parameters of the first potential energy field. This represents the design parameters for the second potential energy field. This represents the design parameters for the third potential energy field. Indicates the design parameters of the attractive potential energy field. This represents the design parameters of the repulsive potential energy field. This indicates the risk coefficient for the first region. This indicates the risk coefficient for the second region. Indicates the first Initial values ​​of the performance function of each follower Indicates the first The final value of the performance function of each follower. Represents the parameters for minimizing the adaptive performance boundary. This represents the parameter for maximizing the adaptive performance boundary. This represents the boundary optimal parameters for adaptive performance. Represents the adaptive performance boundary balance parameter. Represents the parameters of the first adaptive performance boundary variables. This represents the parameters of the second adaptive performance boundary variables. This represents the third adaptive performance boundary variable parameter. This represents the fourth adaptive performance boundary variable parameter. This represents the fifth adaptive performance boundary variable parameter. This indicates the convergence time of the preset performance function. Indicates the parameters of the first distributed reinforcement learning controller. This represents the parameters of the second distributed reinforcement learning controller. This represents the parameters of the third distributed reinforcement learning controller. This represents the positive design parameters of the neural network of the first recognizer. The positive design parameter representing the first update rate. This represents the positive design parameters of the second-recognition neural network. The positive design parameter representing the second update rate. This represents the first positive design parameter of the first actor neural network. This represents the second positive design parameter of the first actor neural network. This represents the first positive design parameter of the first critic neural network. This represents the first positive design parameter of the second actor neural network. This represents the second positive design parameter of the second actor neural network. This represents the first positive design parameter of the second critic neural network.

[0231] Figure 4 The formation of drones and unmanned vehicles guided by the virtual leader's target trajectory and mapping trajectory. axis, shaft and The three-dimensional formation trajectory of the axis demonstrates the drone's climb-cruise-descent and the control process of the unmanned vehicle performing formation obstacle avoidance and passing through narrow passages under the guidance of the drone. Figure 5 Demonstrating drone and unmanned vehicle formations along shaft and The two-dimensional formation trajectory of the axis can be observed from the figure. The performance of the formation of drones and unmanned vehicles through areas with different risk levels and the formation trajectory effect when passing through narrow passages under the protection of privacy can be seen. Figure 6 The trajectories of the drone and unmanned vehicle (UAV) formations moving in different directions are described under the proposed privacy protection mechanism. It can be observed that during the passage through a narrow passage (25-32 seconds), the positions and heading angles of the drone and UAV formations exhibit good convergence. Specifically, the magnified views in the upper right corner of each subplot clearly show that within this interval, due to the intervention of the privacy protection mechanism, the state trajectories of each drone and UAV maintain an overall consistent trend while closely following the preset trajectory of the leader drone. Figure 7 and Figure 8 The formation trajectory errors of drones and unmanned vehicles under the action of non-vulnerable performance functions are shown respectively. From Figure 7 and Figure 8 It can be seen that when the formation tracking error caused by the active obstacle avoidance deviation when encountering obstacles and narrow passages approaches the specified upper limit, the method trades safety by temporarily relaxing the error constraint on the reference path. Figures 9 to 11 The study demonstrates the trajectory effects in the actor weights, critic weights, and recognizer weight norms of the position and attitude systems of UAVs and autonomous vehicles. The rapid convergence of the weights indicates that the designed reinforcement learning update law can efficiently capture system dynamics and approximate the optimal control law in real time. The study also proves that the algorithm exhibits good online learning efficiency and numerical stability when handling UAV and autonomous vehicle systems with nonlinear dynamics. Figure 12 The evolution trajectory of the cost function of accumulated error and control energy consumption during system operation is shown, which will eventually stabilize around a certain value. This indicates that accumulated error and energy consumption will reach an optimal balance point at some point in the future. Specific Implementation Example 3:

[0233] This disclosure also provides an embodiment:

[0234] An electronic device includes: a storage medium and a processing unit; wherein the storage medium is used to store a computer program, and the processing unit exchanges data with the storage medium to execute the computer program during obstacle avoidance control of an air-to-ground heterogeneous unmanned swarm formation, performing the steps of the method described in Specific Embodiment 1. Specific Implementation Example 4:

[0236] A computer-readable storage medium storing a computer program; when the computer program is run, it performs the steps of the method as described in Specific Embodiment 1.

[0237] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0238] The above disclosure only discloses a few specific implementation scenarios. However, this disclosure is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this disclosure.

Claims

1. A method for traffic management and formation obstacle avoidance control of heterogeneous unmanned swarms based on reinforcement learning, characterized in that, include: Establish dynamic models for both unmanned aerial vehicles (UAVs) and unmanned vehicles. A phased privacy protection mechanism for air-to-ground heterogeneous unmanned clusters is obtained based on the aforementioned UAV dynamics model and unmanned vehicle dynamics model. The integration error of the air-ground heterogeneous unmanned swarm formation is determined by using a composite guiding vector field that integrates attractive and repulsive fields. Construct a non-fragile performance constraint function, and obtain an error transformation function based on the ensemble error; then determine the distributed reinforcement learning virtual controller and the actual controller based on the error transformation function. Based on the distributed reinforcement learning virtual controller and the actual controller, the formation integration error converges to the neighborhood of zero, thereby realizing the time-consistent formation control of the air-ground heterogeneous unmanned cluster.

2. The air-to-ground heterogeneous unmanned swarm traffic management and formation obstacle avoidance control method based on reinforcement learning according to claim 1, characterized in that, The phased privacy protection mechanism for obtaining air-to-ground heterogeneous unmanned swarms based on the UAV dynamics model and the unmanned vehicle dynamics model includes: Based on the staged privacy protection function of the air-ground heterogeneous unmanned cluster, obtain the hyperbolic tangent mask coordinate transformation based on the leader and followers; Based on the hyperbolic tangent mask coordinate transformation based on leaders and followers, a staged privacy protection mechanism based on leaders and followers is constructed to flexibly configure the protection window according to task requirements, preventing malicious nodes from deducing the initial position and velocity trajectory of other nodes through state interaction.

3. The air-to-ground heterogeneous unmanned swarm traffic management and formation obstacle avoidance control method based on reinforcement learning according to claim 2, characterized in that, The expression for the hyperbolic tangent mask coordinate transformation based on leaders and followers is: ； ； in, and Represents the phased privacy protection function for the i-th follower; and Represents the phased privacy protection function for the leader; and This represents the weighting factor of the hyperbolic tangent mask function used by the follower; and This represents the weighting factor of the hyperbolic tangent mask function used by the leader; and This indicates the status information of a follower under covert communication. and This indicates the leader's status information under covert communication. and This indicates the follower's status information before the phased privacy protection mechanism takes effect; and This indicates the leader's status information prior to the implementation of the phased privacy protection mechanism. Indicates the time of the control process.

4. The air-to-ground heterogeneous unmanned swarm traffic management and formation obstacle avoidance control method based on reinforcement learning according to claim 2, characterized in that, The phased privacy protection mechanism based on the leader-follower structure is represented as follows: ； ； in, This represents the uncertain variable of the i-th follower after masking; Let represent the unknown bounded perturbation variable of the i-th follower after masking; This represents the control gain of the i-th follower after masking. This represents the control input of the i-th follower after masking. This represents the nonlinear term of the i-th follower after masking. This represents the control signal of the i-th follower after masking. This represents the uncertain variables of the virtual leader after masking. This represents the unknown bounded perturbation variable of the virtual leader after masking. This indicates the control signals of the virtual leader after masking. The first derivative represents the state information of the follower under covert communication; The second derivative represents the state information of the follower under covert communication; This represents the first derivative of the leader's state information under covert communication.

5. The air-to-ground heterogeneous unmanned swarm traffic management and formation obstacle avoidance control method based on reinforcement learning according to claim 2, characterized in that, The construction of the non-fragile performance constraint function and the acquisition of the error transformation function based on the integration error include: Determine the non-fragile performance constraint function based on the preset performance function and the non-fragile adaptive boundary variables; Based on the formation integration error and the non-fragile performance constraint function, an error transformation function is generated to allow for flexible deviations and obtain a safe corridor; The non-fragile performance constraint function is expressed as follows: ； in, Indicates the first Position integration error of a quadcopter drone; Indicates the first Attitude integration error of a quadcopter drone; Indicates the first The integrated error of the position and heading angle of the unmanned vehicle; Indicates the first The lower boundary of the position integration error constraint for a quadcopter UAV; Indicates the first The upper boundary of the positional integrated error constraint for a quadcopter UAV; Indicates the first The lower boundary of the attitude integration error constraint for a quadcopter UAV; Indicates the first The upper boundary of the attitude integration error constraint for a quadcopter UAV; Indicates the first The lower boundary of the integrated error constraint between the position and heading angle of the autonomous vehicle; Indicates the first The upper boundary of the integrated error constraint between the position and heading angle of the unmanned vehicle.

6. The method for traffic management and formation obstacle avoidance control of heterogeneous unmanned swarms based on reinforcement learning according to claim 1, characterized in that: The distributed reinforcement learning virtual controller is represented as: ； ； in, This represents the communication topology of the k-th quadcopter UAV in the position direction; This represents the communication topology of the k-th quadcopter UAV in the attitude direction; This represents the communication topology of the i-th unmanned vehicle in terms of position and heading angle. Indicates the parameters of the first distributed reinforcement learning controller; Indicates the parameters of the second distributed reinforcement learning controller; Indicates the parameters of the third distributed reinforcement learning controller; This represents the precondition of the transformed derivative of the position and orientation error of the k-th quadcopter UAV; This represents the precondition of the transformed derivative of the attitude direction error for the k-th quadcopter UAV; This represents the precondition of the error transformation derivative of the i-th unmanned vehicle in the position and heading angle directions; Let represent the error transformation function in the position direction of the k-th quadcopter UAV; Let represent the error transformation function in the attitude direction of the k-th quadcopter UAV; Let represent the error transformation function of the i-th unmanned vehicle in terms of position and heading angle; Let denote the derivative of the first formation variable of the composite guidance vector field of the r-th quadrotor UAV; This represents the neural network input signal of the position direction of the k-th quadcopter UAV; This represents the input signal of the neural network for the attitude orientation of the k-th quadcopter UAV. This represents the neural network input signal of the i-th autonomous vehicle in terms of position and heading angle. The neural network represents the first identifier in the position direction of the k-th quadcopter UAV. The neural network representing the first recognizer of the k-th quadcopter UAV in the attitude direction; The neural network representing the first identifier of the i-th autonomous vehicle in terms of position and heading angle; This represents the first adaptive law of the k-th quadcopter UAV in the position direction; This represents the first adaptive rate of the k-th quadrotor UAV in the attitude direction; This represents the first adaptive rate of the i-th unmanned vehicle in terms of position and heading angle. This represents the optimal weight estimate of the neural network for the first actor in the position direction of the k-th quadcopter UAV. This represents the optimal weight estimate of the first actor neural network for the k-th quadrotor UAV in the attitude direction; This represents the optimal weight estimate of the neural network for the first actor of the i-th autonomous vehicle in terms of position and heading angle. This represents the first input signal of the neural network in the position direction of the k-th quadcopter UAV; This represents the first input signal of the neural network for the k-th quadcopter UAV in the attitude direction; This represents the first input signal of the neural network for the i-th unmanned vehicle in terms of position and heading angle. Let the first Gaussian function of the k-th quadrotor UAV be oriented in the position direction; Let the first Gaussian function of the k-th quadrotor UAV be oriented in the position direction; Let represent the first Gaussian function of the i-th unmanned vehicle in terms of position and heading angle; The actual controller for distributed reinforcement learning is represented as: ； ； ； in, This represents the control gain of the k-th quadcopter UAV in the position direction; This represents the control gain of the k-th quadcopter UAV in the attitude direction; This represents the control gain of the i-th unmanned vehicle in terms of position and heading angle. Let represent the error variable in the position direction of the k-th quadcopter UAV. Let $\mathbf{k}$ represent the error variable in the attitude direction of the $k$-th quadcopter UAV. Let represent the error variables of the i-th unmanned vehicle in terms of position and heading angle; The second-order neural network represents the position and orientation of the k-th quadcopter UAV. The second-order neural network represents the attitude direction of the k-th quadcopter UAV. The second-recognition neural network represents the position and heading angle of the i-th autonomous vehicle. This represents the parameters of the fourth distributed reinforcement learning controller; This represents the second adaptive law of the k-th quadcopter UAV in the position direction; This represents the second adaptive rate of the k-th quadrotor UAV in the attitude direction; This represents the second adaptive rate of the i-th unmanned vehicle in terms of position and heading angle; This represents the optimal weight estimate of the second actor neural network for the k-th quadcopter UAV in the position direction; This represents the optimal weight estimate of the second actor neural network for the k-th quadcopter UAV in the attitude direction; This represents the optimal weight estimate of the second actor neural network for the i-th unmanned vehicle in terms of position and heading angle. This represents the second input signal of the neural network in the position direction of the k-th quadcopter UAV; This represents the second input signal of the neural network for the k-th quadcopter UAV in the attitude direction; This represents the second input signal of the neural network for the i-th autonomous vehicle in terms of position and heading angle. This represents the second Gaussian function of the k-th quadrotor UAV in the position direction; Denotes the second Gaussian function of the k-th quadrotor UAV in the attitude direction; Let represent the second Gaussian function of the i-th unmanned vehicle in terms of position and heading angle.

7. The method for traffic management and formation obstacle avoidance control of heterogeneous unmanned swarms based on reinforcement learning according to claim 1, characterized in that, The implementation of time-consistent formation control for the air-to-ground heterogeneous unmanned cluster includes: By analyzing using Lyapunov stability theory, we prove that the overall Lyapunov function expression is: ； in, Let represent the set of all stability coefficients of the first and second Lyapunov functions; Let represent the remaining terms of the first and second Lyapunov functions.

8. The method for traffic management and formation obstacle avoidance control of heterogeneous unmanned swarms based on reinforcement learning according to claim 1, characterized in that: The expression for the composite guiding vector field is: ； in, Indicates the number of drones; Indicates the number of driverless cars; This indicates the total number of drones and unmanned vehicles; This represents the first design parameter of the attraction-guided vector field; Represents the attraction-guided vector field; This represents the first design parameter of the repulsive force guiding vector field; This represents a vector field guided by repulsive force.

9. A reinforcement learning-based air-to-ground heterogeneous unmanned swarm formation obstacle avoidance control system, based on the reinforcement learning-based air-to-ground heterogeneous unmanned swarm traffic control and formation obstacle avoidance control method described in any one of claims 1-8, characterized in that, include: The module includes a virtual leader module, an external and internal collision module, a non-fragile preset performance function module, a performance index module, a module for the r-th UAV and the j-th unmanned vehicle, a composite guiding vector field module, an error transformation function module, an HJB equation module, a recognition neural network update rate module, an actor neural network update rate module, a critic neural network update rate module, a reinforcement learning neural network module, a model uncertainty and external disturbance module, a virtual controller module, and an actual controller module. The virtual leader module interacts with the composite guidance vector field module to generate the desired position and speed information of the pilot drone, and transmits the obtained desired position and speed information of the pilot drone to the composite guidance vector field module. The external and internal collision module interacts with the composite guidance vector field module to generate position information of static obstacles, position information of the r-th UAV and position information of the j-th unmanned vehicle, and transmits the obtained position information of static obstacles, position information of the r-th UAV and position information of the j-th unmanned vehicle to the composite guidance vector field module. The nonfragile preset performance function module interacts with the error transformation function module and the performance index module to generate the upper and lower boundaries of the nonfragile performance constraints, and then transmits the obtained upper and lower boundaries of the nonfragile performance constraints to the error transformation function module and the performance index module. The r-th UAV and j-th unmanned vehicle module interacts with the composite guidance vector field module to obtain model uncertainty and uncertainty input from the external disturbance module, disturbance information and actual control signal input from the actual controller module, generate position and velocity information of the r-th UAV and j-th unmanned vehicle, and transmit the obtained position and velocity information of the r-th UAV and j-th unmanned vehicle to the composite guidance vector field module. The composite guidance vector field module interacts with the error conversion function module to obtain the position and velocity information of the r-th UAV and the j-th unmanned vehicle and the position information of the static obstacle, generate the composite guidance force of the r-th UAV and the j-th unmanned vehicle, and transmit the obtained composite guidance force of the r-th UAV and the j-th unmanned vehicle to the error conversion function module. The error conversion function module interacts with the performance index module to obtain the upper and lower boundaries of the composite guiding force and non-fragile performance constraints of the r-th UAV and the j-th unmanned vehicle, generate the error signals of the r-th UAV and the j-th unmanned vehicle, and transmit the obtained error signals of the r-th UAV and the j-th unmanned vehicle to the performance index module. The performance index module interacts with the HJB equation module to generate a first performance evaluation index from the error signals of the r-th UAV and j-th unmanned vehicle input by the non-fragile performance constraint upper and lower boundaries and the error transformation function module, and then transmits the obtained first performance evaluation index to the HJB equation module. The HJB equation module interacts with the recognition neural network update rate module, the actor neural network update rate module, and the critic neural network update rate module to generate the first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation, and then transmits the obtained first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation to the recognition neural network update rate module, the actor neural network update rate module, and the critic neural network update rate module, respectively. The identifier neural network update rate module interacts with the reinforcement learning neural network module to obtain the first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation, generate the first identifier neural network weight matrix and the second identifier neural network weight matrix, and transmit the obtained first identifier neural network weight matrix and the second identifier neural network weight matrix to the reinforcement learning neural network module. The actor neural network update rate module interacts with the reinforcement learning neural network module to obtain the first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation, generate the first actor neural network weight matrix and the second actor neural network weight matrix, and pass the obtained first actor neural network weight matrix and the second actor neural network weight matrix to the reinforcement learning neural network module. The critic neural network update rate module interacts with the reinforcement learning neural network module to obtain the first Hamilton-Jacobi-Bellman equation and the second Hamilton-Jacobi-Bellman equation, generate the first critic neural network weight matrix and the second critic neural network weight matrix, and pass the obtained first critic neural network weight matrix and second critic neural network weight matrix to the reinforcement learning neural network module. The reinforcement learning neural network module obtains the weight matrices of the first and second recognizer neural networks, the weight matrices of the first and second actor neural networks, the weight matrices of the first and second critic neural networks, the weight matrices of the first and second critic neural networks, generates approximate values ​​of unknown terms, and transmits the obtained approximate values ​​of unknown terms to the actual controller module. The virtual controller module interacts with the reinforcement learning neural network module to obtain approximate values ​​of unknown terms, generate virtual control signals, error variables, and a second performance evaluation index, and then transmits the obtained virtual control signals, error variables, and second performance evaluation index to the HJB equation module. The actual controller module interacts with the r-th UAV and j-th unmanned vehicle modules to obtain approximate values ​​of unknown terms, generate actual control signals, and transmit the obtained actual control signals to the r-th UAV and j-th unmanned vehicle modules.

10. The air-to-ground heterogeneous unmanned swarm formation obstacle avoidance control system based on reinforcement learning according to claim 9, characterized in that: The model uncertainty and external disturbance module generates uncertainty and disturbance information, and transmits the obtained uncertainty and disturbance information to the r-th UAV and j-th unmanned vehicle module.

Images