Multi-unmanned aerial vehicle optimal game limited acceleration reinforcement learning control method and device

Abstract

The invention provides a multi-unmanned aerial vehicle optimal game limited accelerated reinforcement learning control method and device, and relates to the technical field of accelerated reinforcement learning. The method comprises the following steps: constructing an evaluation neural network, and approaching a performance index function of a hyperbolic tangent function and a game neighbor item, the optimal limited control input of the unmanned aerial vehicle and the limited control input of the unmanned aerial vehicle under the worst condition; based on the input parameters, constructing a Hamilton-Jacobian error equation; through an error equation, constructing a summation square error which contains current information and past information and is provided with adjusting parameters; designing a weight updating law for evaluating the neural network; calculating the weight of the next iteration according to a weight updating law; and solving an error between two adjacent iteration weights, comparing the error with a preset threshold value, if the error is smaller than the preset threshold value, stopping iteration, and outputting a solution approaching the optimal game consistency control problem of the distributed unmanned aerial vehicle system. According to the invention, the cooperation efficiency between the unmanned aerial vehicles can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of accelerated reinforcement learning algorithms, and in particular to a multi-UAV optimal game restricted accelerated reinforcement learning control method and device. Background Art

[0002] Consensus control of multi-agent systems is the most fundamental coordination method. For a swarm of drones, it ensures that each drone's information, such as position, speed, and heading, converges, providing the foundation for subsequent formation, distributed decision-making, and collaborative planning. Multi-UAV distributed control offers excellent scalability and robustness, enabling efficient coordination under local communication conditions, reducing communication and computational burdens, and improving the system's real-time performance and fault tolerance. Multi-UAV distributed optimal game consensus control addresses the problem of optimal coordinated control of multiple drones under competition, collaboration, or resource constraints. Combining consensus and game theory, it enables drones to reach consensus and optimize individual or overall goals when faced with resource conflicts or multi-objective tasks. For example, when multiple drones perform search and rescue and supply delivery missions in a disaster area, cooperative game allocation is used to maximize mission efficiency, avoid collisions, and conserve energy. This prevents multiple drones from traveling to the same target, and consensus control maintains formation consistency or path coordination en route to the target. In practical applications, drone actuator capabilities, system design, safety requirements, or environmental conditions impose hard or soft constraints on control inputs, including thrust, speed, and angular velocity. For example, a group of drones must form a formation to navigate a narrow valley or urban corridor. The narrow passage limits maximum lateral acceleration, preventing rapid turns and resulting in input constraints. Therefore, it is essential to consider input constraints when designing optimal game-consistent control. Reinforcement learning is a method that enables intelligent agents to learn optimal behavioral strategies through interaction with the environment, guided by reward signals. Reinforcement learning enables multiple drones in an unknown environment to learn optimal strategies through interaction with each other and the environment, ultimately achieving consistent control and a Nash equilibrium for the mission objective.

[0003] To address the problem of optimal game-consistent control of multiple UAVs, a series of reinforcement learning-related methods have been proposed. Among these, a data-based, off-policy reinforcement learning algorithm, combining reinforcement learning algorithms with the asymmetric input-constrained optimal synchronization problem of heterogeneous, unknown, nonlinear multi-agent systems, is proposed to learn solutions to the constrained Hamilton-Jacobi-Bellman equation without requiring dynamic information about the agents. However, the proposed reinforcement learning algorithm's convergence speed needs to be improved, and it fails to consider the game relationships between UAVs, which can lead to inefficient system coordination, resource waste, policy conflicts, and unpredictable individual behavior. In practical multi-UAV control, especially in environments with task competition, limited energy, and restricted communication, introducing game modeling is key to achieving efficient, robust, and scalable coordinated control. Therefore, developing an optimal game-constrained consensus control method for distributed multi-UAV systems that accelerates reinforcement learning is a core issue that researchers in this field urgently need to address. Summary of the Invention

[0004] In order to solve the technical problems of poor scalability, input constraints, and low coordination efficiency of drone consensus methods in the existing technology, the embodiments of the present invention provide a multi-drone optimal game restricted accelerated reinforcement learning control method and device. The technical solution is as follows:

[0005] On the one hand, a multi-UAV optimal game restricted accelerated reinforcement learning control method is provided. The method is implemented by a multi-UAV optimal game restricted accelerated reinforcement learning control device, and the method includes:

[0006] S1. Construct a judgment neural network and initialize the weights of the judgment neural network;

[0007] S2. Based on the judgment neural network, the performance index function with hyperbolic tangent function and game neighbor term, the optimal restricted control input of the UAV and the restricted control input of the UAV in the worst case are approximated;

[0008] S3. Constructing a Hamilton-Jacobi error equation based on the performance indicator function, the optimal restricted control input of the UAV, and the restricted control input of the UAV under the worst case;

[0009] S4. Based on the Hamilton-Jacobi error equation, construct a sum-squared error with adjustment parameters that includes the current UAV state information and the past UAV state information;

[0010] S5. Designing a weight update law for evaluating the neural network based on the summed square error with adjustment parameters; calculating the latest weight for the next iteration based on the weight update law for evaluating the neural network;

[0011] S6. Calculate the error between the weights of the evaluation neural network of the current iteration and the weights of the next iteration; compare the error with a preset threshold; if the error is less than the preset threshold, stop the iteration and output the weights of the evaluation neural network of the current iteration; wherein the weights of the evaluation neural network of the current iteration are close to the solution of the optimal game consensus control problem of the distributed unmanned aerial vehicle system.

[0012] On the other hand, a multi-UAV optimal game restricted accelerated reinforcement learning control device is provided, which is applied to the multi-UAV optimal game restricted accelerated reinforcement learning control method, and the device includes:

[0013] The first construction unit is used to construct a judgment neural network and initialize the weights of the judgment neural network;

[0014] An acquisition unit, used to approximate a performance index function with a hyperbolic tangent function and a game neighbor term, an optimal restricted control input of the UAV, and a restricted control input of the UAV in the worst case based on a judgment neural network;

[0015] a second construction unit, configured to construct a Hamilton-Jacobi error equation according to the performance indicator function, an optimal restricted control input of the UAV, and a restricted control input of the UAV under a worst-case scenario;

[0016] The third construction unit is used to construct a sum-squared error with adjustment parameters including current UAV state information and past UAV state information according to the Hamilton-Jacobi error equation;

[0017] A design unit is used to design a weight update law for evaluating the neural network based on the summed square error with adjustment parameters; and calculate the latest weight for the next iteration based on the weight update law for evaluating the neural network;

[0018] The output unit is used to solve the error between the evaluation neural network weight of the current iteration and the weight of the next iteration; the error is compared with a preset threshold. If the error is less than the preset threshold, the iteration is stopped and the evaluation neural network weight of the current iteration is output; wherein the evaluation neural network weight of the current iteration is a solution that approximates the optimal game consensus control problem of the distributed unmanned aerial vehicle system.

[0019] On the other hand, a multi-UAV optimal game constrained accelerated reinforcement learning control device is provided, which includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned multi-UAV optimal game constrained accelerated reinforcement learning control methods is implemented.

[0020] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned multi-UAV optimal game constrained accelerated reinforcement learning control methods.

[0021] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0022] The embodiment of the present invention first constructs a judgment neural network and initializes the weights of the judgment neural network; based on the judgment neural network, a performance index function with a hyperbolic tangent function and a game neighbor term, the optimal restricted control input of the drone, and the restricted control input of the drone in the worst case are approximated; secondly, a Hamilton-Jacobi error equation is constructed according to the performance index function, the optimal restricted control input of the drone, and the restricted control input of the drone in the worst case; based on the Hamilton-Jacobi error equation, a sum-squared error with adjustment parameters containing current drone state information and past drone state information is constructed; based on the sum-squared error with adjustment parameters, a weight update law of the judgment neural network is designed; finally, based on the weight update law of the judgment neural network, the latest weight of the next iteration is calculated; the error between the weight of the judgment neural network of the current iteration and the weight of the next iteration is solved; the error is compared with a preset threshold, and if the error is less than the preset threshold, the iteration is stopped and the weight of the judgment neural network of the current iteration is output; wherein the weight of the judgment neural network of the current iteration is a solution that approximates the optimal game consistent control problem of the distributed drone system.

[0023] An embodiment of the present invention proposes an optimal game-constrained consensus control method for a distributed multi-UAV system based on accelerated reinforcement learning. Compared with the existing reinforcement learning consensus control algorithm, the advantage of the method proposed in the embodiment of the present invention is that it takes into account the game relationship between UAVs, improves the collaboration efficiency and scalability, and the algorithm converges quickly without the need to meet continuous incentive conditions. The constant weight update law of the neural network is designed and judged by utilizing current and past accumulated information. The update law embeds a positive adjustment parameter to improve the flexibility of the algorithm, and the Nash equilibrium solution of the optimal game-constrained consensus control problem of the distributed multi-UAV system is obtained through continuous iteration. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1This is a flow chart of a multi-UAV optimal game restricted accelerated reinforcement learning control method provided by an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of a specific implementation process structure of a multi-UAV optimal game restricted accelerated reinforcement learning control method provided by an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of a specific implementation process of a multi-UAV optimal game restricted accelerated reinforcement learning control method provided by an embodiment of the present invention;

[0028] Figure 4 This is a structural diagram of a multi-UAV optimal game restricted accelerated reinforcement learning control device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0030] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0031] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0032] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0033] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0034] The embodiment of the present invention provides a multi-UAV optimal game restricted accelerated reinforcement learning control method, which can be implemented by a multi-UAV optimal game restricted accelerated reinforcement learning control device, which can be a terminal or a server. Figure 1The flowchart of the multi-UAV optimal game restricted accelerated reinforcement learning control method is shown. The processing flow of the method may include the following steps:

[0035] S1. Construct a judgment neural network and initialize the weights of the judgment neural network.

[0036] In a feasible implementation, the tracking reference signal of the UAV system is expressed by the following formula (1):

[0037] (1)

[0038] in, represents the reference signal; Represents the UAV system dynamic matrix, where Represents the state dimension of the reference signal.

[0039] In a feasible implementation, the UAV i is modeled based on the tracking reference signal of the UAV system, wherein the modeling process is expressed by the following formula (2):

[0040] (2)

[0041] in, Indicates drone The state input; drones Control input; represents the dimension of the control input; Represents the dynamic matrix of the drone; represents the input dynamic matrix; Indicates the control input elements; Indicates drone The control input is restricted by the bounds that satisfy .

[0042] Among them, the weights of the evaluation neural network of each drone are initialized, and the basis function of the evaluation neural network is designed.

[0043] S2. Based on the judgment neural network, the performance index function with hyperbolic tangent function and game neighbor term, the optimal restricted control input of the UAV and the restricted control input of the UAV in the worst case are approximated.

[0044] Optionally, the specific implementation process of S2 is expressed by the following formulas (3)-(5):

[0045] (3)

[0046] (4)

[0047] (5)

[0048] in, represents the estimated value of the performance indicator function; Indicates the initialization of the weights of the judgment neural network; represents the basis function; represents the relative state error of UAV i; Indicates drone The optimal game restricted control estimate of ; represents the optimal game restricted control estimate of UAV j; tanh represents the hyperbolic tangent function; represents a symmetric positive definite matrix; Representation matrix The inverse of is a matrix The transpose of Represents the basis function about The partial derivative of yes The transpose of It's a drone Control input restricted bounds; It is the boundary where the drone i control input is limited; is a symmetric positive definite matrix The inverse, It's a drone Input dynamic matrix The transpose of is the attenuation level.

[0049] Among them, based on the relative relationship between drones, the relative state error dynamics of drone i is obtained.

[0050] Optionally, the relative state error of UAV i is expressed by the following formula (6):

[0051] (6)

[0052] in, represents the relative state error of UAV i; Indicates drone The number of in-degree neighbors of ; Representatives from drones to drones There exists a directed path, otherwise, ; Representative drone You can get reference tracking information, otherwise, ; It's a drone The input dynamic matrix, It's a drone control input.

[0053] S3. Construct the Hamilton-Jacobi error equation based on the performance index function, the optimal restricted control input of the UAV, and the restricted control input of the UAV in the worst case.

[0054] Alternatively, the Hamilton-Jacobi error equation is expressed by the following formula (7):

[0055] (7)

[0056] in, represents the Hamilton-Jacobi error; Indicates a pre-set time; represents a symmetric positive definite matrix; express The transpose of Indicates about control function, It is the boundary where the drone’s control input is limited. represents a symmetric positive definite matrix; Indicates about control function, It's a drone Control input is restricted by bounds, Indicates drone A symmetric positive definite matrix; Represents a basis function at time and The difference, Indicates time The basis function values ​​under ; Indicates time The basis function values ​​under ; Indicates that from the drone to drones Directed paths of Indicates drone The number of in-degree neighbors of Indicates drone Directed paths between references to follow; Represents the square value of the attenuation level; N represents the number of drones; represents the integral variable; Represents the weights of the neural network.

[0057] S4. According to the Hamilton-Jacobi error equation, a sum-squared error with adjustment parameters is constructed, which includes the current UAV state information and the past UAV state information.

[0058] Among them, the drone status information includes: position, speed and attitude parameters.

[0059] Optionally, the summed square error with adjustment parameters including current information and past information is expressed by the following formula (8):

[0060] (8)

[0061] in, is the sum-squared error; is the Hamilton-Jacobi error The transpose of is 1 to The value of is a positive constant, is the first adjustment parameter, It's time The error value below, It's time The difference below.

[0062] S5. Based on the summed square error with adjustment parameters, design a weight update law for evaluating the neural network; based on the weight update law for evaluating the neural network, calculate the latest weight for the next iteration.

[0063] Among them, according to the sum square error with adjustment parameters, the gradient descent method is used to design the weight update law of the evaluation neural network; among them, according to the weight update law of the evaluation neural network, the process of calculating the latest weight of the next iteration does not need to meet the continuous excitation condition.

[0064] Optionally, the weight update law of the evaluation neural network is expressed by the following formula (9):

[0065] (9)

[0066] in, Represents the derivative of the weight of the judging neural network; Represents the basis function at time and The difference between represents the second adjustment parameter; Indicates time The error value below: is a positive constant; is the first adjustment parameter; It's time The difference under represents the Hamilton-Jacobi error.

[0067] S6. Calculate the error between the weights of the current iteration's judgment neural network and the weights of the next iteration; compare the error with a preset threshold; if the error is less than the preset threshold, stop the iteration and output the weights of the current iteration's judgment neural network; the weights of the current iteration's judgment neural network are close to the solution to the optimal game consensus control problem of the distributed UAV system.

[0068] If the error is greater than or equal to the preset threshold, the iteration is continued, and the number of iterations is increased by 1, and steps S2-S5 are repeated.

[0069] Optionally, the process of comparing the error with a preset threshold is expressed by the following formula (10):

[0070] (10)

[0071] in, Indicates the preset threshold; Represents the weight of the judgment neural network for the next iteration; Represents the weight of the current iteration's judgment neural network.

[0072] Among them, such as Figure 2 It is a schematic diagram of the specific implementation process structure of a multi-UAV optimal game restricted accelerated reinforcement learning control method provided by an embodiment of the present invention; in a feasible implementation method, a judgment neural network is constructed, the judgment network weights are initialized, and the judgment neural network basis functions are designed; based on the judgment neural network, a performance index function with a hyperbolic tangent function and a game neighbor term, an optimal restricted control input of the UAV, and a restricted control input of the UAV in the worst case are approximated; according to the performance index function with a hyperbolic tangent function and a game neighbor term, an optimal restricted control input of the UAV, and a restricted control input of the UAV in the worst case, a Hamilton-Jacobi error equation is constructed; according to the Hamilton-Jacobi error equation, a sum-squared error with adjustment parameters containing current information and past information is obtained; based on The method comprises the following steps: a first step is to solve the weight error of two adjacent iterations and determine whether the error is less than a preset threshold. If the error is greater than or equal to the preset threshold, the number of iterations is increased by 1, and the process of approximating the performance index function with hyperbolic tangent function and game neighbor terms, the optimal restricted control input of the UAV, and the restricted control input of the UAV in the worst case is repeated until the calculated weight error of two adjacent iterations is less than the preset threshold. The iteration is stopped and the Nash equilibrium solution of the optimal game restricted consistent control problem of the UAV is output.

[0073] The embodiment of the present invention first constructs a judgment neural network and initializes the weights of the judgment neural network; based on the judgment neural network, a performance index function with a hyperbolic tangent function and a game neighbor term, the optimal restricted control input of the drone, and the restricted control input of the drone in the worst case are approximated; secondly, a Hamilton-Jacobi error equation is constructed according to the performance index function, the optimal restricted control input of the drone, and the restricted control input of the drone in the worst case; based on the Hamilton-Jacobi error equation, a sum-squared error with adjustment parameters containing current drone state information and past drone state information is constructed; based on the sum-squared error with adjustment parameters, a weight update law of the judgment neural network is designed; finally, based on the weight update law of the judgment neural network, the latest weight of the next iteration is calculated; the error between the weight of the judgment neural network of the current iteration and the weight of the next iteration is solved; the error is compared with a preset threshold, and if the error is less than the preset threshold, the iteration is stopped and the weight of the judgment neural network of the current iteration is output; wherein the weight of the judgment neural network of the current iteration is a solution that approximates the optimal game consistent control problem of the distributed drone system.

[0074] An embodiment of the present invention proposes an optimal game-constrained consensus control method for a distributed multi-UAV system based on accelerated reinforcement learning. Compared with the existing reinforcement learning consensus control algorithm, the advantage of the method proposed in the embodiment of the present invention is that it takes into account the game relationship between UAVs, improves the collaboration efficiency and scalability, and the algorithm converges quickly without the need to meet continuous incentive conditions. The constant weight update law of the neural network is designed and judged by utilizing current and past accumulated information. The update law embeds a positive adjustment parameter to improve the flexibility of the algorithm, and the Nash equilibrium solution of the optimal game-constrained consensus control problem of the distributed multi-UAV system is obtained through continuous iteration.

[0075] Figure 3 This is a block diagram of a multi-UAV optimal game restricted accelerated reinforcement learning control device according to an exemplary embodiment. The device is used in a multi-UAV optimal game restricted accelerated reinforcement learning control method. Figure 3 The device includes a first construction unit 310, an acquisition unit 320, a second construction unit 330, a third construction unit 340, a design unit 350, and an output unit 360.

[0076] A first construction unit 310 is used to construct a judgment neural network and initialize the weights of the judgment neural network;

[0077] An acquisition unit 320 is configured to approximate a performance indicator function with a hyperbolic tangent function and a game neighbor term, an optimal restricted control input of the UAV, and a restricted control input of the UAV in a worst-case scenario based on a judgment neural network;

[0078] A second construction unit 330 is configured to construct a Hamilton-Jacobi error equation based on the performance indicator function, the optimal restricted control input of the UAV, and the restricted control input of the UAV under the worst case;

[0079] The third construction unit 340 is configured to construct a sum-squared error with adjustment parameters including current UAV state information and past UAV state information according to the Hamilton-Jacobi error equation;

[0080] A design unit 350 is configured to design a weight update law for evaluating the neural network based on the summed square error with adjustment parameters; and calculate the latest weight for the next iteration based on the weight update law for evaluating the neural network;

[0081] Output unit 360 is used to solve the error between the evaluation neural network weight of the current iteration and the weight of the next iteration; compare the error with a preset threshold, and if the error is less than the preset threshold, stop the iteration and output the evaluation neural network weight of the current iteration; wherein the evaluation neural network weight of the current iteration is a solution that approximates the optimal game consensus control problem of the distributed unmanned aerial vehicle system.

[0082] Optionally, the process of approximating the performance index function with the hyperbolic tangent function and the game neighbor term, the optimal restricted control input of the UAV, and the restricted control input of the UAV in the worst case based on the judgment neural network is expressed by the following formulas (1)-(3):

[0083] (1)

[0084] (2)

[0085] (3)

[0086] in, represents the estimated value of the performance indicator function; Indicates the initialization of the weights of the judgment neural network; represents the basis function; represents the relative state error of UAV i; Indicates drone The optimal game restricted control estimate of ; represents the optimal game restricted control estimate of UAV j; tanh represents the hyperbolic tangent function; represents a symmetric positive definite matrix; Representation matrix The inverse of is a matrix The transpose of Represents the basis function about The partial derivative of yes The transpose of It's a drone Control input restricted bounds; It is the boundary where the drone i control input is limited; is a symmetric positive definite matrix The inverse, It's a drone Input dynamic matrix The transpose of is the attenuation level.

[0087] Optionally, the relative state error of the UAV i is expressed by the following formula (4):

[0088] (4)

[0089] in, represents the relative state error of UAV i; Indicates drone The number of in-degree neighbors of ; Representatives from drones to drones There exists a directed path, otherwise, ; Representative drone You can get reference tracking information, otherwise, ; It's a drone The input dynamic matrix, It's a drone control input.

[0090] Optionally, the Hamilton-Jacobi error equation is expressed by the following formula (5):

[0091] (5)

[0092] in, represents the Hamilton-Jacobi error; Indicates a pre-set time; represents a symmetric positive definite matrix; express The transpose of Indicates about control function, It is the boundary where the drone’s control input is limited. represents a symmetric positive definite matrix; Indicates about control function, It's a drone Control input is restricted by bounds, Indicates drone A symmetric positive definite matrix; Represents a basis function at time and The difference, Indicates time The basis function values ​​under ; Indicates time The basis function values ​​under ; Indicates that from the drone to drones Directed paths of Indicates drone The number of in-degree neighbors of Indicates drone Directed paths between references to follow; Represents the square value of the attenuation level; N represents the number of drones; represents the integral variable; Represents the weights of the neural network.

[0093] Optionally, the summed square error with adjustment parameters including current information and past information is expressed by the following formula (6):

[0094] (6)

[0095] in, is the sum-squared error; is the Hamilton-Jacobi error The transpose of is 1 to The value of is a positive constant, is the first adjustment parameter, It's time The error value below, It's time The difference below.

[0096] Optionally, the weight update law of the evaluation neural network is expressed by the following formula (7):

[0097] (7)

[0098] in, Represents the derivative of the weight of the judging neural network; Represents the basis function at time and The difference between represents the second adjustment parameter; Indicates time The error value below: is a positive constant; is the first adjustment parameter; It's time The difference under represents the Hamilton-Jacobi error.

[0099] Optionally, the process of comparing the error with a preset threshold is expressed by the following formula (8):

[0100] (8)

[0101] in, Indicates the preset threshold; Represents the weight of the judgment neural network for the next iteration; Represents the weight of the current iteration's judgment neural network.

[0102] The embodiment of the present invention first constructs a judgment neural network and initializes the weights of the judgment neural network; based on the judgment neural network, a performance index function with a hyperbolic tangent function and a game neighbor term, the optimal restricted control input of the drone, and the restricted control input of the drone in the worst case are approximated; secondly, a Hamilton-Jacobi error equation is constructed according to the performance index function, the optimal restricted control input of the drone, and the restricted control input of the drone in the worst case; based on the Hamilton-Jacobi error equation, a sum-squared error with adjustment parameters containing current drone state information and past drone state information is constructed; based on the sum-squared error with adjustment parameters, a weight update law of the judgment neural network is designed; finally, based on the weight update law of the judgment neural network, the latest weight of the next iteration is calculated; the error between the weight of the judgment neural network of the current iteration and the weight of the next iteration is solved; the error is compared with a preset threshold, and if the error is less than the preset threshold, the iteration is stopped and the weight of the judgment neural network of the current iteration is output; wherein the weight of the judgment neural network of the current iteration is a solution that approximates the optimal game consistent control problem of the distributed drone system.

[0103] The present invention overcomes the poor scalability, input constraints, and low collaborative efficiency of multi-UAV consensus methods. By introducing current and past information into the Hamilton-Jacobi error, the present invention avoids the need for continuous excitation conditions. Game theory is used to maximize collaborative efficiency, and inverse hyperbolic functions are used to process input constraints, transforming constrained problems into unconstrained ones. This allows for rapid convergence of optimal game-constrained consistent Nash equilibrium solutions for multi-UAV systems.

[0104] An embodiment of the present invention proposes an optimal game-constrained consensus control method for a distributed multi-UAV system based on accelerated reinforcement learning. Compared with the existing reinforcement learning consensus control algorithm, the advantage of the method proposed in the embodiment of the present invention is that it takes into account the game relationship between UAVs, improves the collaboration efficiency and scalability, and the algorithm converges quickly without the need to meet continuous incentive conditions. The constant weight update law of the neural network is designed and judged by utilizing current and past accumulated information. The update law embeds a positive adjustment parameter to improve the flexibility of the algorithm, and the Nash equilibrium solution of the optimal game-constrained consensus control problem of the distributed multi-UAV system is obtained through continuous iteration.

[0105] Figure 4 is a structural diagram of a multi-UAV optimal game restricted accelerated reinforcement learning control device provided by an embodiment of the present invention, such as Figure 4 As shown, the multi-UAV optimal game restricted accelerated reinforcement learning control device may include the above Figure 3 The multi-UAV optimal game restricted accelerated reinforcement learning control device shown. Optionally, the multi-UAV optimal game restricted accelerated reinforcement learning control device 410 may include a first processor 2001.

[0106] Optionally, the multi-UAV optimal game restricted accelerated reinforcement learning control device 410 may further include a memory 2002 and a transceiver 2003 .

[0107] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0108] The following combination Figure 4 The components of the multi-UAV optimal game restricted accelerated reinforcement learning control device 410 are specifically introduced:

[0109] The first processor 2001 is the control center of the multi-UAV optimal game restricted accelerated reinforcement learning control device 410 and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs) or one or more field programmable gate arrays (FPGAs).

[0110] Optionally, the first processor 2001 can execute various functions of the multi-UAV optimal game restricted accelerated reinforcement learning control device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0111] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0112] In a specific implementation, as an embodiment, the multi-UAV optimal game restricted accelerated reinforcement learning control device 410 may also include multiple processors, such as Figure 4 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0113] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0114] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and accessed through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0115] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0116] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0117] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or can exist independently and be controlled through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0118] It should be noted that Figure 4 The structure of the multi-UAV optimal game restricted accelerated reinforcement learning control device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0119] In addition, the technical effects of the multi-UAV optimal game constrained accelerated reinforcement learning control device 410 can refer to the technical effects of the multi-UAV optimal game constrained accelerated reinforcement learning control method described in the above method embodiment, and will not be repeated here.

[0120] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0121] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0122] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0123] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0124] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0125] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0126] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0127] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0128] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0129] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0130] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0131] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0132] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A multi-UAV optimal game restricted accelerated reinforcement learning control method, characterized by: The method comprises: S1. Construct a judgment neural network and initialize the weights of the judgment neural network; S2. Based on the judgment neural network, the performance index function with hyperbolic tangent function and game neighbor term, the optimal restricted control input of the UAV and the restricted control input of the UAV in the worst case are approximated; S3. Constructing a Hamilton-Jacobi error equation based on the performance indicator function, the optimal restricted control input of the UAV, and the restricted control input of the UAV under the worst case; S4. Based on the Hamilton-Jacobi error equation, construct a sum-squared error with adjustment parameters that includes the current UAV state information and the past UAV state information; S5. Designing a weight update law for evaluating the neural network based on the summed square error with adjustment parameters; calculating the latest weight for the next iteration based on the weight update law for evaluating the neural network; S6. Calculate the error between the evaluation neural network weights of the current iteration and the evaluation neural network weights of the next iteration; compare the error with a preset threshold; if the error is less than the preset threshold, stop the iteration and output the evaluation neural network weights of the current iteration; wherein the evaluation neural network weights of the current iteration are close to the solution of the optimal game consensus control problem of the distributed unmanned aerial vehicle system.

2. The multi-UAV optimal game restricted accelerated reinforcement learning control method according to claim 1 is characterized in that: The process of S2 based on the judgment neural network to approximate the performance index function with hyperbolic tangent function and game neighbor term, the optimal restricted control input of the UAV and the restricted control input of the UAV in the worst case is expressed by the following formulas (1)-(3): （1） （2） （3） in, represents the estimated value of the performance indicator function; Indicates the initialization of the weights of the judgment neural network; represents the basis function; represents the relative state error of UAV i; Indicates drone The optimal game restricted control estimate of ; represents the optimal game restricted control estimate of UAV j; tanh represents the hyperbolic tangent function; represents a symmetric positive definite matrix; Representation matrix The inverse of is a matrix The transpose of Represents the basis function about The partial derivative of yes The transpose of It's a drone Control input restricted bounds; It is the boundary where the drone i control input is limited; is a symmetric positive definite matrix The inverse, It's a drone Input dynamic matrix The transpose of is the attenuation level.

3. The multi-UAV optimal game restricted accelerated reinforcement learning control method according to claim 2 is characterized in that: The relative state error of the UAV i is expressed by the following formula (4): （4） in, represents the relative state error of UAV i; Indicates drone The number of in-degree neighbors of ; Representatives from drones to drones There exists a directed path, otherwise, ; Representative drone You can get reference tracking information, otherwise, ; It's a drone The input dynamic matrix, It's a drone control input.

4. The multi-UAV optimal game restricted accelerated reinforcement learning control method according to claim 1, characterized in that: The Hamilton-Jacobi error equation is expressed by the following formula (5): （5） in, represents the Hamilton-Jacobi error; Indicates a pre-set time; represents a symmetric positive definite matrix; express The transpose of Indicates about control function, It is the boundary where the drone’s control input is limited. represents a symmetric positive definite matrix; Indicates about control function, It's a drone Control input is restricted by bounds, Indicates drone A symmetric positive definite matrix; Represents a basis function at time and The difference, Indicates time The basis function values ​​under ; Indicates time The basis function values ​​under ; From the drone to drones Directed paths of Indicates drone The number of in-degree neighbors of Indicates drone Directed paths between references to follow; Represents the square value of the attenuation level; N represents the number of drones; represents the integral variable; Represents the weights of the neural network.

5. The multi-UAV optimal game restricted accelerated reinforcement learning control method according to claim 1 is characterized in that: The summed square error with adjustment parameters including current information and past information is expressed by the following formula (6): （6） in, is the sum-squared error; is the Hamilton-Jacobi error The transpose of is 1 to The value of is a positive constant, is the first adjustment parameter, It's time The error value below, It's time The difference below.

6. The multi-UAV optimal game restricted accelerated reinforcement learning control method according to claim 1, characterized in that: The weight update law of the evaluation neural network is expressed by the following formula (7): （7） in, Represents the derivative of the weight of the judging neural network; Represents the basis function at time and The difference between represents the second adjustment parameter; Indicates time The error value below: is a positive constant; is the first adjustment parameter; It's time The difference under represents the Hamilton-Jacobi error.

7. The multi-UAV optimal game restricted accelerated reinforcement learning control method according to claim 1 is characterized in that: The process of comparing the error with a preset threshold is expressed by the following formula (8): （8） in, Indicates the preset threshold; Represents the weight of the judgment neural network for the next iteration; Represents the weight of the current iteration's judgment neural network.

8. A multi-UAV optimal game-constrained accelerated reinforcement learning control device, the multi-UAV optimal game-constrained accelerated reinforcement learning control device is used to implement the multi-UAV optimal game-constrained accelerated reinforcement learning control method according to any one of claims 1 to 7, characterized in that: The device comprises: The first construction unit is used to construct a judgment neural network and initialize the weights of the judgment neural network; An acquisition unit, used to approximate a performance index function with a hyperbolic tangent function and a game neighbor term, an optimal restricted control input of the UAV, and a restricted control input of the UAV in the worst case based on a judgment neural network; a second construction unit, configured to construct a Hamilton-Jacobi error equation according to the performance indicator function, an optimal restricted control input of the UAV, and a restricted control input of the UAV under a worst-case scenario; The third construction unit is used to construct a sum-squared error with adjustment parameters including current UAV state information and past UAV state information according to the Hamilton-Jacobi error equation; A design unit is used to design a weight update law for evaluating the neural network based on the summed square error with adjustment parameters; and calculate the latest weight for the next iteration based on the weight update law for evaluating the neural network; The output unit is used to solve the error between the evaluation neural network weight of the current iteration and the weight of the next iteration; the error is compared with a preset threshold. If the error is less than the preset threshold, the iteration is stopped and the evaluation neural network weight of the current iteration is output; wherein the evaluation neural network weight of the current iteration is a solution that approximates the optimal game consensus control problem of the distributed unmanned aerial vehicle system.

9. A multi-UAV optimal game restricted accelerated reinforcement learning control device, characterized in that: The multi-UAV optimal game restricted accelerated reinforcement learning control device includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images