Multi-uav optimal game restricted accelerated reinforcement learning control method and device

Abstract

The application provides a multi-unmanned aerial vehicle optimal game restricted accelerated reinforcement learning control method and device, and relates to the technical field of accelerated reinforcement learning. The method comprises the following steps: constructing a judgment neural network, approximating a performance index function with a hyperbolic tangent function and a game neighbor term, an optimal restricted control input of an unmanned aerial vehicle, and a restricted control input of the unmanned aerial vehicle in a worst case; based on the input parameters, a Hamilton-Jacobi error equation is constructed; through the error equation, a sum-of-squares error with an adjusting parameter containing current information and past information is constructed; a weight updating law of the judgment neural network is designed; according to the weight updating law, the weight of the next iteration is calculated; the error of the weights of two adjacent iterations is solved, and is compared with a preset threshold value; if the error is smaller than the preset threshold value, the iteration is stopped, and the solution approximating the optimal game consistent control problem of a distributed unmanned aerial vehicle system is output. The application can improve the cooperation efficiency between unmanned aerial vehicles.

Description

Technical Field

[0001] This invention relates to the field of accelerated reinforcement learning algorithm technology, and in particular to a constrained accelerated reinforcement learning control method and device for multi-UAV optimal game. Background Technology

[0002] Consistent control in multi-agent systems is the most fundamental coordination method. For drone swarms, it ensures that the information of each drone, such as position, speed, and heading, tends to be consistent, providing a foundation for subsequent formation, distributed decision-making, and collaborative planning. Distributed control of multiple drones possesses good scalability and robustness, enabling efficient collaboration under localized communication conditions, reducing communication and computational burdens, and improving the system's real-time performance and fault tolerance. Distributed optimal game-theoretic consistent control of multiple drones addresses the optimal collaborative control problem under conditions of competition, cooperation, or resource constraints. Combining consistency and game theory, it enables drones to reach consensus and optimize individual or overall objectives when facing resource conflicts or multiple objectives. For example, when multiple drones perform search and rescue and material delivery missions in disaster areas, to maximize mission efficiency, avoid collisions, and save energy, cooperative game theory is used to allocate task points, i.e., avoiding multiple drones going to the same target, and maintaining formation consistency or path coordination during the journey to the target through consistent control. In practical applications, the actuator capabilities, system design, safety requirements, or environmental conditions of drones impose hard or soft constraints on control inputs, including thrust, velocity, and angular velocity. For example, when a group of drones needs to form a path through a narrow valley or urban corridor, the maximum lateral acceleration is limited, preventing rapid turns and resulting in input constraints. Therefore, it is essential to consider input constraints when designing optimal game-theoretic consistent control. Reinforcement learning is a method that enables agents to learn optimal behavioral strategies by interacting with their environment and guided by reward signals. Reinforcement learning allows multiple drones in an unknown environment to learn optimal strategies through interaction with each other and the environment, ultimately achieving consistent control and game-theoretic Nash equilibrium under the mission objective.

[0003] To address the optimal game-theoretic consensus control problem for multi-UAV systems, a series of reinforcement learning-related methods have been proposed. Among them, a data-based non-policy reinforcement learning algorithm is proposed to learn solutions to the constrained Hamiltonian-Jacobi-Bellman equations for heterogeneous, unknown, nonlinear multi-agent systems, considering the asymmetric input-constrained optimal synchronization problem. This algorithm does not require dynamic information from the agents. However, the convergence speed of the proposed reinforcement learning algorithm needs improvement, and it does not consider the game-theoretic relationships between UAVs, leading to low system coordination efficiency, resource waste, policy conflicts, and unpredictable individual behavior. In practical multi-UAV control, especially in environments with task competition, limited energy, and restricted communication, introducing game modeling is crucial for achieving efficient, robust, and scalable cooperative control. Therefore, how to construct an optimal game-theoretic consensus control method for distributed multi-UAV systems that accelerates reinforcement learning is one of the core problems that urgently needs to be solved by researchers in this field. Summary of the Invention

[0004] To address the technical problems of poor scalability, input constraints, and low collaborative efficiency in existing UAV consensus methods, this invention provides a constrained accelerated reinforcement learning control method and apparatus for multi-UAV optimal game theory. The technical solution is as follows:

[0005] On the one hand, a constrained accelerated reinforcement learning control method for multi-UAV optimal game is provided. This method is implemented by a multi-UAV optimal game constrained accelerated reinforcement learning control device, and includes:

[0006] S1. Construct the evaluation neural network and initialize the weights of the evaluation neural network;

[0007] S2. Based on the evaluation neural network, approximate the performance index function with hyperbolic tangent function and game neighbor term, the optimal constrained control input of UAV and the constrained control input of UAV in the worst case;

[0008] S3. Based on the performance index function, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst case, construct the Hamilton-Jacobi error equation;

[0009] S4. Based on the Hamilton-Jacobi error equation, construct a summation squared error with adjustable parameters that includes current UAV state information and past UAV state information.

[0010] S5. Based on the summation squared error with adjustment parameters, design the weight update law of the evaluation neural network; based on the weight update law of the evaluation neural network, calculate the latest weights for the next iteration.

[0011] S6. Solve for the error between the weights of the current iteration and the weights of the next iteration; compare the error with a preset threshold. If the error is less than the preset threshold, stop the iteration and output the weights of the current iteration; wherein the weights of the current iteration are the solution to the optimal game-theoretic consistent control problem of the distributed unmanned aerial vehicle system.

[0012] On the other hand, a constrained accelerated reinforcement learning control device for multi-UAV optimal game is provided. This device is applied to the constrained accelerated reinforcement learning control method for multi-UAV optimal game. The device includes:

[0013] The first building unit is used to build the evaluation neural network and initialize the weights of the evaluation neural network;

[0014] The acquisition unit is used to approximate the performance index function with hyperbolic tangent function and game neighbor term, the optimal constrained control input of the UAV and the constrained control input of the UAV in the worst case, based on the evaluation neural network;

[0015] The second construction unit is used to construct the Hamilton-Jacobi error equation based on the performance index function, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst case.

[0016] The third building unit is used to construct a summation squared error with adjustable parameters, which includes current UAV state information and past UAV state information, based on the Hamilton-Jacobi error equation.

[0017] The design unit is used to design the weight update law of the evaluation neural network based on the summation squared error with adjustment parameters; and to calculate the latest weights for the next iteration based on the weight update law of the evaluation neural network.

[0018] The output unit is used to calculate the error between the weights of the current iteration and the weights of the next iteration; compare the error with a preset threshold, and if the error is less than the preset threshold, stop the iteration and output the weights of the current iteration; wherein the weights of the current iteration are the solution to the optimal game-theoretic consistent control problem of the distributed unmanned aerial vehicle system.

[0019] On the other hand, a multi-UAV optimal game constrained accelerated reinforcement learning control device is provided, the multi-UAV optimal game constrained accelerated reinforcement learning control device comprising: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for multi-UAV optimal game constrained accelerated reinforcement learning control.

[0020] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods of multi-UAV optimal game-constrained accelerated reinforcement learning control methods.

[0021] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0022] This invention first constructs an evaluation neural network and initializes its weights. Based on the evaluation neural network, it approximates a performance index function with a hyperbolic tangent function and game neighbor terms, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst-case scenario. Next, based on the performance index function, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst-case scenario, it constructs a Hamilton-Jacobi error equation. Based on the Hamilton-Jacobi error equation, it constructs a summation squared error with adjustment parameters, including current UAV state information and past UAV state information. Based on the summation squared error with adjustment parameters, it designs a weight update law for the evaluation neural network. Finally, based on the weight update law of the evaluation neural network, it calculates the latest weights for the next iteration. It solves for the error between the current iteration's evaluation neural network weights and the next iteration's weights. It compares the error with a preset threshold; if the error is less than the preset threshold, it stops iterating and outputs the current iteration's evaluation neural network weights. The current iteration's evaluation neural network weights approximate the solution to the optimal game-theoretic consistent control problem of a distributed UAV system.

[0023] This invention proposes an optimal game-theoretic constrained consensus control method for distributed multi-UAV systems based on accelerated reinforcement learning. Compared with existing reinforcement learning consensus control algorithms, the proposed method has the advantage of considering the game relationship between UAVs, improving cooperation efficiency and scalability. Furthermore, the algorithm converges quickly and does not require satisfying continuous incentive conditions. It utilizes current and past accumulated information to design a constant weight update law for the evaluation neural network. This update law embeds a positive adjustment parameter, enhancing the algorithm's flexibility. Through continuous iteration, the Nash equilibrium solution to the optimal game-theoretic constrained consensus control problem of distributed multi-UAV systems is obtained. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1This is a flowchart of a multi-UAV optimal game-constrained accelerated reinforcement learning control method provided by an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram illustrating the specific implementation process of a multi-UAV optimal game-constrained accelerated reinforcement learning control method provided in this embodiment of the invention.

[0027] Figure 3 This is a schematic diagram illustrating the specific implementation process of a multi-UAV optimal game-constrained accelerated reinforcement learning control method provided in this embodiment of the invention;

[0028] Figure 4 This is a schematic diagram of the structure of a multi-UAV optimal game-constrained accelerated reinforcement learning control device provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0030] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0031] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0032] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0033] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0034] This invention provides a multi-UAV optimal game constrained accelerated reinforcement learning control method. This method can be implemented by a multi-UAV optimal game constrained accelerated reinforcement learning control device, which can be a terminal or a server. Figure 1The flowchart shown is for a multi-UAV optimal game-constrained accelerated reinforcement learning control method. The processing flow of this method may include the following steps:

[0035] S1. Construct the evaluation neural network and initialize the weights of the evaluation neural network.

[0036] In one feasible implementation, the tracking reference signal of the unmanned aerial vehicle system is represented by the following formula (1):

[0037] (1)

[0038] in, Indicates a reference signal; Represents the dynamic matrix of the unmanned aerial vehicle system, where, The state dimension represents the reference signal.

[0039] In one feasible implementation, the UAV i is modeled based on the tracking reference signal of the UAV system, wherein the modeling process is represented by the following formula (2):

[0040] (2)

[0041] in, Indicates drone Status input; drones Control input; Indicates the dimension controlling the input; The dynamic matrix representing the UAV; Indicates the input dynamic matrix; The first control input One element; Indicates drone Control input bounds, satisfying .

[0042] This involves initializing the weights of the evaluation neural network for each drone and designing the basis functions of the evaluation neural network.

[0043] S2. Based on the evaluation neural network, approximate the performance index function with hyperbolic tangent function and game neighbor term, the optimal constrained control input of UAV and the constrained control input of UAV in the worst case.

[0044] Optionally, the specific implementation process of S2 is represented by the following formulas (3)-(5):

[0045] (3)

[0046] (4)

[0047] (5)

[0048] in, This represents the estimated value of the performance index function; This indicates the initialization of the evaluation neural network weights; Describe the basis functions; This represents the relative state error of drone i; Indicates drone The optimal game-constrained control estimate; tanh represents the optimal game-constrained control estimate of UAV j; tanh represents the hyperbolic tangent function. Represents a symmetric positive definite matrix; Representation matrix The reverse; It is a matrix Transpose of; Indicate the basis functions with respect to The partial derivative; yes Transpose of; It is a drone Control input limits; This is the boundary of the restricted control input for the drone; It is a symmetric positive definite matrix The reverse, It is a drone Input dynamic matrix transpose, It refers to the attenuation level.

[0049] Among them, the relative state error dynamics of UAV i are obtained based on the relative relationship between UAVs.

[0050] Optionally, the relative state error of UAV i is expressed by the following formula (6):

[0051] (6)

[0052] in, This represents the relative state error of drone i; Indicates drone The number of in-degree neighbors; Representative from drones To drones There exists a directed path; otherwise, ; Representing drones You can obtain reference and follow information; otherwise, ; It is a drone The input dynamic matrix, It is a drone Control input.

[0053] S3. Based on the performance index function, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst case, construct the Hamilton-Jacobi error equation.

[0054] Alternatively, the Hamilton-Jacobi error equation can be expressed by the following formula (7):

[0055] (7)

[0056] in, This represents the Hamilton-Jacobi error; Indicates a pre-set time; Represents a symmetric positive definite matrix; express Transpose of; Indicates about control The function, This is the boundary of the limited control input for drones. Represents a symmetric positive definite matrix; Indicates about control The function, It is a drone Control input limits, Indicates drone A symmetric positive definite matrix; Describe a basis function in time and The difference, Indicates time The basis function values; Indicates time Basis function values; Indicates from drones To drones A directed path; Indicates drone The number of in-degree neighbors; Indicates drone The directed path between reference following; The value represents the square of the attenuation level; N represents the number of drones. Represents the integral variable; This represents the weights used to evaluate the neural network.

[0057] S4. Based on the Hamilton-Jacobi error equation, construct a summation squared error with adjustable parameters that includes current UAV state information and past UAV state information.

[0058] The drone's status information includes its position, speed, and attitude parameters.

[0059] Optionally, the summation squared error of the adjusted parameter, which includes current and past information, is expressed by the following formula (8):

[0060] (8)

[0061] in, It is the summation square error; Hamilton-Jacobi error Transpose of; It is 1 to The value, It is a positive constant. It is the first adjustment parameter. It is time The error value below, It is time The difference below.

[0062] S5. Based on the summation squared error with adjustment parameters, design the weight update law for the evaluation neural network; based on the weight update law of the evaluation neural network, calculate the latest weights for the next iteration.

[0063] Specifically, based on the summation squared error with adjustment parameters, the gradient descent method is used to design the weight update law of the evaluation neural network; and the process of calculating the latest weight for the next iteration based on the weight update law of the evaluation neural network does not need to satisfy the continuous excitation condition.

[0064] Alternatively, the weight update law of the evaluation neural network can be expressed by the following formula (9):

[0065] (9)

[0066] in, The derivative representing the weights of a neural network; Indicates the basis functions at time and The difference; This indicates the second adjustment parameter; Indicates time The error value below; It is a positive constant; It is the first adjustment parameter; It is time The difference below; This represents the Hamilton-Jacobi error.

[0067] S6. Solve for the error between the weights of the current iteration and the weights of the next iteration; compare the error with a preset threshold. If the error is less than the preset threshold, stop the iteration and output the weights of the current iteration; where the weights of the current iteration are the solution to the optimal game-theoretic consistent control problem of the distributed unmanned aerial vehicle system.

[0068] If the error is greater than or equal to the preset threshold, continue iterating and increment the iteration count by 1, repeating steps S2-S5.

[0069] Optionally, the process of comparing the error with a preset threshold is represented by the following formula (10):

[0070] (10)

[0071] in, Indicates the preset threshold; This represents the weights of the evaluation neural network for the next iteration; This represents the weights of the evaluation neural network in the current iteration.

[0072] Among them, such as Figure 2 This is a schematic diagram of the specific implementation process of a multi-UAV optimal game-theoretic constrained accelerated reinforcement learning control method provided by an embodiment of the present invention. In one feasible implementation, a judgment neural network is constructed, the weights of the judgment network are initialized, and the basis functions of the judgment neural network are designed. Based on the judgment neural network, a performance index function with a hyperbolic tangent function and game neighbor terms, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst-case scenario are approximated. Based on the performance index function with a hyperbolic tangent function and game neighbor terms, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst-case scenario, a Hamilton-Jacobi error equation is constructed. Based on the summation squared error of the Hamilton-Jacobi error equation, which includes current and past information with adjustable parameters, the basis functions are... The algorithm calculates the summation squared error with adjustable parameters, which includes current and past information. It then constructs a weight update law for a judgment neural network. This law is used to calculate the weights for the next iteration. The algorithm solves for the weight error between two adjacent iterations and determines if the error is less than a preset threshold. If the error is greater than or equal to the preset threshold, the iteration count is incremented by 1. The process of approximating the performance index function with the hyperbolic tangent function and the game's neighbor term, the optimal constrained control input of the UAV, and the worst-case constrained control input of the UAV is repeated until the calculated weight error between two adjacent iterations is less than the preset threshold. At this point, the iteration stops, and the Nash equilibrium solution to the optimal game-theoretic constrained consistent control problem of the UAV is output.

[0073] This invention first constructs an evaluation neural network and initializes its weights. Based on the evaluation neural network, it approximates a performance index function with a hyperbolic tangent function and game neighbor terms, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst-case scenario. Next, based on the performance index function, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst-case scenario, it constructs a Hamilton-Jacobi error equation. Based on the Hamilton-Jacobi error equation, it constructs a summation squared error with adjustment parameters, including current UAV state information and past UAV state information. Based on the summation squared error with adjustment parameters, it designs a weight update law for the evaluation neural network. Finally, based on the weight update law of the evaluation neural network, it calculates the latest weights for the next iteration. It solves for the error between the current iteration's evaluation neural network weights and the next iteration's weights. It compares the error with a preset threshold; if the error is less than the preset threshold, it stops iterating and outputs the current iteration's evaluation neural network weights. The current iteration's evaluation neural network weights approximate the solution to the optimal game-theoretic consistent control problem of a distributed UAV system.

[0074] This invention proposes an optimal game-theoretic constrained consensus control method for distributed multi-UAV systems based on accelerated reinforcement learning. Compared with existing reinforcement learning consensus control algorithms, the proposed method has the advantage of considering the game relationship between UAVs, improving cooperation efficiency and scalability. Furthermore, the algorithm converges quickly and does not require satisfying continuous incentive conditions. It utilizes current and past accumulated information to design a constant weight update law for the evaluation neural network. This update law embeds a positive adjustment parameter, enhancing the algorithm's flexibility. Through continuous iteration, the Nash equilibrium solution to the optimal game-theoretic constrained consensus control problem of distributed multi-UAV systems is obtained.

[0075] Figure 3 This is a block diagram illustrating a multi-UAV optimal game constrained accelerated reinforcement learning control device according to an exemplary embodiment. The device is used in a multi-UAV optimal game constrained accelerated reinforcement learning control method. (Refer to...) Figure 3 The device includes a first construction unit 310, an acquisition unit 320, a second construction unit 330, a third construction unit 340, a design unit 350, and an output unit 360. Wherein:

[0076] The first building unit 310 is used to build the evaluation neural network and initialize the weights of the evaluation neural network;

[0077] The acquisition unit 320 is used to approximate the performance index function with hyperbolic tangent function and game neighbor term, the optimal constrained control input of the UAV and the constrained control input of the UAV in the worst case based on the evaluation neural network;

[0078] The second construction unit 330 is used to construct the Hamilton-Jacobi error equation based on the performance index function, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst case.

[0079] The third building unit 340 is used to build a summation squared error with adjustable parameters, which includes current UAV state information and past UAV state information, based on the Hamilton-Jacobi error equation.

[0080] Design unit 350 is used to design the weight update law of the evaluation neural network based on the summation squared error with adjustment parameters; and to calculate the latest weights for the next iteration based on the weight update law of the evaluation neural network.

[0081] Output unit 360 is used to calculate the error between the weights of the current iteration and the weights of the next iteration; compare the error with a preset threshold, and if the error is less than the preset threshold, stop the iteration and output the weights of the current iteration; wherein the weights of the current iteration are the solution to the optimal game-theoretic consistent control problem of the distributed unmanned aerial vehicle system.

[0082] Optionally, the process of approximating the performance index function with hyperbolic tangent function and game neighbor term, the optimal constrained control input of the UAV and the constrained control input of the UAV in the worst case based on the evaluation neural network is expressed by the following formulas (1)-(3):

[0083] (1)

[0084] (2)

[0085] (3)

[0086] in, This represents the estimated value of the performance index function; This indicates the initialization of the evaluation neural network weights; Describe the basis functions; This represents the relative state error of drone i; Indicates drone The optimal game-constrained control estimate; tanh represents the optimal game-constrained control estimate of UAV j; tanh represents the hyperbolic tangent function. Represents a symmetric positive definite matrix; Representation matrix The reverse; It is a matrix Transpose of; Indicate the basis functions with respect to The partial derivative; yes Transpose of; It is a drone Control input limits; This is the boundary of the restricted control input for the drone; It is a symmetric positive definite matrix The reverse, It is a drone Input dynamic matrix transpose, It refers to the attenuation level.

[0087] Optionally, the relative state error of the UAV i is expressed by the following formula (4):

[0088] (4)

[0089] in, This represents the relative state error of drone i; Indicates drone The number of in-degree neighbors; Representative from drones To drones There exists a directed path; otherwise, ; Representing drones You can obtain reference and follow information; otherwise, ; It is a drone The input dynamic matrix, It is a drone Control input.

[0090] Optionally, the Hamilton-Jacobi error equation is expressed by the following formula (5):

[0091] (5)

[0092] in, This represents the Hamilton-Jacobi error; Indicates a pre-set time; Represents a symmetric positive definite matrix; express Transpose of; Indicates about control The function, This is the boundary of the limited control input for drones. Represents a symmetric positive definite matrix; Indicates about control The function, It is a drone Control input limits, Indicates drone A symmetric positive definite matrix; Describe a basis function in time and The difference, Indicates time Basis function values; Indicates time Basis function values; Indicates from drones To drones A directed path; Indicates drone The number of in-degree neighbors; Indicates drone The directed path between reference following; The value represents the square of the attenuation level; N represents the number of drones. Represents the integral variable; This represents the weights used to evaluate the neural network.

[0093] Optionally, the summation squared error of the adjusted parameter, which includes current and past information, is expressed by the following formula (6):

[0094] (6)

[0095] in, It is the summation square error; Hamilton-Jacobi error Transpose of; It is 1 to The value, It is a positive constant. It is the first adjustment parameter. It is time The error value below, It is time The difference below.

[0096] Optionally, the weight update law of the evaluation neural network is expressed by the following formula (7):

[0097] (7)

[0098] in, The derivative representing the weights of a neural network; Indicates the basis functions at time and The difference; This indicates the second adjustment parameter; Indicates time The error value below; It is a positive constant; It is the first adjustment parameter; It is time The difference below; This represents the Hamilton-Jacobi error.

[0099] Optionally, the process of comparing the error with a preset threshold is represented by the following formula (8):

[0100] (8)

[0101] in, Indicates the preset threshold; This represents the weights of the evaluation neural network for the next iteration; This represents the weights of the evaluation neural network in the current iteration.

[0102] This invention first constructs an evaluation neural network and initializes its weights. Based on the evaluation neural network, it approximates a performance index function with a hyperbolic tangent function and game neighbor terms, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst-case scenario. Next, based on the performance index function, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst-case scenario, it constructs a Hamilton-Jacobi error equation. Based on the Hamilton-Jacobi error equation, it constructs a summation squared error with adjustment parameters, including current UAV state information and past UAV state information. Based on the summation squared error with adjustment parameters, it designs a weight update law for the evaluation neural network. Finally, based on the weight update law of the evaluation neural network, it calculates the latest weights for the next iteration. It solves for the error between the current iteration's evaluation neural network weights and the next iteration's weights. It compares the error with a preset threshold; if the error is less than the preset threshold, it stops iterating and outputs the current iteration's evaluation neural network weights. The current iteration's evaluation neural network weights approximate the solution to the optimal game-theoretic consistent control problem of a distributed UAV system.

[0103] The embodiments of this invention overcome the problems of poor scalability, input constraints, and low collaborative efficiency in multi-UAV consensus methods. By introducing current and past information into the Hamilton-Jacobi error, the embodiments of this invention avoid the need for continuous excitation conditions. Collaborative efficiency is maximized using game theory, and the input constraints are handled using an inverse hyperbolic function, transforming the constrained problem into an unconstrained one. The embodiments of this invention achieve rapid convergence of the optimal game-theoretic constrained consensus Nash equilibrium solution for multi-UAV systems.

[0104] This invention proposes an optimal game-theoretic constrained consensus control method for distributed multi-UAV systems based on accelerated reinforcement learning. Compared with existing reinforcement learning consensus control algorithms, the proposed method has the advantage of considering the game relationship between UAVs, improving cooperation efficiency and scalability. Furthermore, the algorithm converges quickly and does not require satisfying continuous incentive conditions. It utilizes current and past accumulated information to design a constant weight update law for the evaluation neural network. This update law embeds a positive adjustment parameter, enhancing the algorithm's flexibility. Through continuous iteration, the Nash equilibrium solution to the optimal game-theoretic constrained consensus control problem of distributed multi-UAV systems is obtained.

[0105] Figure 4 This is a schematic diagram of the structure of a multi-UAV optimal game-constrained accelerated reinforcement learning control device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the multi-UAV optimal game-constrained accelerated reinforcement learning control device may include the above-mentioned Figure 3 The multi-UAV optimal game constrained accelerated reinforcement learning control device shown is optionally included. The multi-UAV optimal game constrained accelerated reinforcement learning control device 410 may include a first processor 2001.

[0106] Optionally, the multi-UAV optimal game-constrained accelerated reinforcement learning control device 410 may also include a memory 2002 and a transceiver 2003.

[0107] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0108] The following is combined Figure 4 A detailed introduction to each component of the multi-UAV optimal game-constrained accelerated reinforcement learning control device 410 is provided:

[0109] The first processor 2001 is the control center of the multi-UAV optimal game-constrained accelerated reinforcement learning control device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0110] Optionally, the first processor 2001 can execute various functions of the multi-UAV optimal game restricted acceleration reinforcement learning control device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0111] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0112] In a specific implementation, as one example, the multi-UAV optimal game-constrained accelerated reinforcement learning control device 410 may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0113] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0114] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the multi-UAV optimal game-constrained accelerated reinforcement learning control device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0115] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0116] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0117] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the multi-UAV optimal game-constrained accelerated reinforcement learning control device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0118] It should be noted that, Figure 4 The structure of the multi-UAV optimal game-constrained accelerated reinforcement learning control device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0119] Furthermore, the technical effect of the multi-UAV optimal game restricted accelerated reinforcement learning control device 410 can be referred to the technical effect of the multi-UAV optimal game restricted accelerated reinforcement learning control method described in the above method embodiments, and will not be repeated here.

[0120] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0121] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0122] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0123] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0124] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0125] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0126] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0127] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0128] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0131] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A constrained accelerated reinforcement learning control method for optimal game theory among multiple unmanned aerial vehicles (UAVs), characterized in that, The method includes: S1. Construct the evaluation neural network and initialize the weights of the evaluation neural network; S2. Based on the evaluation neural network, approximate the performance index function with hyperbolic tangent function and game neighbor term, the optimal constrained control input of UAV and the constrained control input of UAV in the worst case; The process by which S2 approximates the performance index function with hyperbolic tangent function and game neighbor term, the optimal restricted control input of the UAV and the restricted control input of the UAV in the worst case based on the evaluation neural network is expressed by the following formulas (1)-(3): （1） （2） （3） in, This represents the estimated value of the performance index function; This indicates the initialization of the evaluation neural network weights; Describe the basis functions; This represents the relative state error of drone i; Indicates drone The optimal game-constrained control estimate; tanh represents the optimal game-constrained control estimate of UAV j; tanh represents the hyperbolic tangent function. Represents a symmetric positive definite matrix; Representation matrix The reverse; It is a matrix Transpose of; Indicate the basis functions with respect to The partial derivative; yes Transpose of; It is a drone Control input limits; This is the boundary of the restricted control input for the drone; It is a symmetric positive definite matrix The reverse, It is a drone Input dynamic matrix transpose, It is the attenuation level; The relative state error of the UAV i is expressed by the following formula (4): （4） in, Represents the dynamic matrix of the unmanned aerial vehicle system; This represents the relative state error of drone i; Indicates drone The number of in-degree neighbors; Representative from drones To drones There exists a directed path; otherwise, ; Representing drones You can obtain reference and follow information; otherwise, ; It is a drone The input dynamic matrix, It is a drone Control input; S3. Based on the performance index function, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst case, construct the Hamilton-Jacobi error equation; S4. Based on the Hamilton-Jacobi error equation, construct a summation squared error with adjustable parameters that includes current UAV state information and past UAV state information. S5. Based on the summation squared error with adjustment parameters, design the weight update law of the evaluation neural network; based on the weight update law of the evaluation neural network, calculate the latest weights for the next iteration. S6. Solve for the error between the current iteration's evaluation neural network weights and the next iteration's evaluation neural network weights; compare the error with a preset threshold. If the error is less than the preset threshold, stop the iteration and output the current iteration's evaluation neural network weights; wherein the current iteration's evaluation neural network weights are the solution to the optimal game-theoretic consistent control problem of the distributed unmanned aerial vehicle system.

2. The multi-UAV optimal game-constrained accelerated reinforcement learning control method according to claim 1, characterized in that, The Hamilton-Jacobi error equation is expressed by the following formula (5): （5） in, This represents the Hamilton-Jacobi error; Indicates a pre-set time; Represents a symmetric positive definite matrix; express Transpose of; Indicates about control The function, This is the boundary of the limited control input for drones. Represents a symmetric positive definite matrix; Indicates about control The function, It is a drone Control input limits, Indicates drone A symmetric positive definite matrix; Describe a basis function in time and The difference, Indicates time Basis function values; Indicates time Basis function values; Indicates from drones To drones A directed path; Indicates drone The number of in-degree neighbors; Indicates drone The directed path between reference following; The value represents the square of the attenuation level; N represents the number of drones. Represents the integral variable; This represents the weights used to evaluate the neural network.

3. The multi-UAV optimal game-constrained accelerated reinforcement learning control method according to claim 1, characterized in that, The summation squared error of the adjusted parameter, which includes current and past information, is expressed by the following formula (6): （6） in, It is the summation square error; Hamilton-Jacobi error Transpose of; It is 1 to The value, It is a positive constant. It is the first adjustment parameter. It is time The error value below, It is time The difference below.

4. The multi-UAV optimal game-constrained accelerated reinforcement learning control method according to claim 1, characterized in that, The weight update law of the evaluation neural network is expressed by the following formula (7): （7） in, The derivative representing the weights of a neural network; Indicates the basis functions at time and The difference; This indicates the second adjustment parameter; Indicates time The error value below; It is a positive constant; It is the first adjustment parameter; It is time The difference below; This represents the Hamilton-Jacobi error.

5. The multi-UAV optimal game-constrained accelerated reinforcement learning control method according to claim 1, characterized in that, The process of comparing the error with a preset threshold is represented by the following formula (8): （8） in, Indicates the preset threshold; This represents the weights of the evaluation neural network for the next iteration; This represents the weights of the evaluation neural network in the current iteration.

6. A multi-UAV optimal game constrained accelerated reinforcement learning control device, wherein the multi-UAV optimal game constrained accelerated reinforcement learning control device is used to implement the multi-UAV optimal game constrained accelerated reinforcement learning control method as described in any one of claims 1-5, characterized in that, The device includes: The first building unit is used to build the evaluation neural network and initialize the weights of the evaluation neural network; The acquisition unit is used to approximate the performance index function with hyperbolic tangent function and game neighbor term, the optimal constrained control input of the UAV and the constrained control input of the UAV in the worst case, based on the evaluation neural network; The second construction unit is used to construct the Hamilton-Jacobi error equation based on the performance index function, the optimal constrained control input of the UAV, and the constrained control input of the UAV in the worst case. The third building unit is used to construct a summation squared error with adjustable parameters, which includes current UAV state information and past UAV state information, based on the Hamilton-Jacobi error equation. The design unit is used to design the weight update law of the evaluation neural network based on the summation squared error with adjustment parameters; and to calculate the latest weights for the next iteration based on the weight update law of the evaluation neural network. The output unit is used to calculate the error between the weights of the current iteration and the weights of the next iteration; compare the error with a preset threshold, and if the error is less than the preset threshold, stop the iteration and output the weights of the current iteration; wherein the weights of the current iteration are the solution to the optimal game-theoretic consistent control problem of the distributed unmanned aerial vehicle system.

7. A multi-UAV optimal game-constrained accelerated reinforcement learning control device, characterized in that, The multi-UAV optimal game-constrained accelerated reinforcement learning control device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 5.

Images