Continuous time unmanned cluster system anti-synchronization optimization control method

Abstract

The invention relates to the field of unmanned aerial vehicle control methods, and particularly discloses a continuous time unmanned cluster system anti-synchronization optimization control method, which comprises the following steps: constructing a topological structure and a Laplacian connection matrix of an unmanned cluster system according to connection conditions and cooperative competition conditions among unmanned aerial vehicles of the unmanned cluster system; designing a strategy gradient algorithm fusing event triggering and a priority experience playback mechanism; the method comprises the following steps: constructing an event-critic neural network based on event triggering; the network determines the updating moment of the network weight by utilizing an event triggering mechanism, so that a non-periodic learning process is realized; the Critic network evaluates the approximate control action of the Actor network, the Actor network adjusts the control action according to the evaluation of the Critic network, and the whole process is updated by gradient descent.

Description

Technical Field

[0001] The invention relates to a control method, in particular to an anti-synchronization optimization control method for a continuous-time unmanned cluster system, and belongs to the field of unmanned aerial vehicle control methods. Background Art

[0002] In recent years, experts and scholars have been inspired by the swarming behavior of organisms in nature and applied the consistency of unmanned swarm systems to the collaborative control of complex systems. The consistency problem of unmanned swarm systems has important application prospects in the fields of smart grids, formation control, and intelligent connected vehicles.

[0003] For unmanned cluster systems, the consistency problem is a basic phenomenon. All drones achieve the same state through information exchange under the action of a consistent control protocol. Existing related work mostly considers the cooperative relationship between unmanned cluster systems, but in practical applications, due to resource constraints, drones will compete for limited resources. On the other hand, in order to save communication resources, many literatures have studied the consistency of unmanned clusters based on event triggering, but the existing event triggering conditions are only related to current data. However, in the process of system optimization, historical data contains a lot of learnable information. Only focusing on current data may lead to the omission of valid information in the past, which will have a certain impact on the stability and optimization of the system. The event triggering method of fusion priority experience replay proposed in the present invention selects important historical data in order of priority, so that the event triggering condition depends on the valid historical data. This method aims to effectively balance system performance while making full use of communication resources. Summary of the invention

[0004] The technical difficulty that the present invention aims to solve is that the problem of data utilization is generally ignored in current research work. In fact, drones generate a large amount of data during the execution of tasks, so it is particularly important to improve data utilization. At the same time, in view of the limitations of drones in computing power and storage space, as well as the complexity of the task itself, the present invention proposes an innovative solution - a synchronous optimization control method for unmanned cluster systems based on event triggering mechanism and priority experience playback. This method aims to achieve anti-synchronous optimization control of unmanned cluster systems by efficiently utilizing the data generated by drones while overcoming their hardware resource limitations.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] A continuous-time unmanned cluster system anti-synchronization optimization control method comprises the following steps:

[0007] S1: constructing the topological structure and Laplace connection matrix of the unmanned swarm system according to the connection status and cooperation and competition status between the drones in the unmanned swarm system;

[0008] S2: A policy gradient algorithm that integrates event triggering and priority experience replay mechanism is designed. Based on this, a new local error system is introduced to design a performance indicator function.

[0009] S3: An event-triggered actor-critic neural network is constructed. The network uses the event-triggered mechanism to determine the update time of the network weights, thereby realizing a non-periodic learning process.

[0010] S4: The Critic network evaluates the approximate control actions of the Actor network, and the Actor network adjusts the control actions based on the evaluation of the Critic network. The entire process is updated using gradient descent. A non-periodic update method is used when training the neural network parameters. When the neural network parameters of the Actor-Critic network structure are no longer updated, the approximate optimal consistent control strategy is obtained.

[0011] Optionally, the local error system under event triggering in step S2 is as follows:

[0012]

[0013] in, represents the triggering time of drone i event, and t i,k <t i,k+1 , (k=1,2,...), define the current error state as and It represents the weighted sampling error state, α i Represents the importance sampling weight in the priority experience replay strategy.

[0014] Optionally, the leader UAV and the follower UAV in step S2 are updated using the corresponding leader dynamic equation and follower dynamic equation; and the local state error system of the UAV is reconstructed according to the leader dynamic equation and the follower dynamic equation, and the performance function is determined based on the local state error system of the UAV, wherein:

[0015] The dynamic equation of the leader drone is expressed as:

[0016]

[0017] Where x0(t)∈R n Indicates the position status of the leader drone;

[0018] The dynamic equation of the follower drone is expressed as:

[0019]

[0020] Among them, x i (t)∈R n and Represent the position state and control input of follower UAV i, A∈R n×n and Represent the dynamic matrix and feedback gain matrix of the system respectively.

[0021] Optionally, the local tracking error of the drone is defined as:

[0022]

[0023] in, represents the local state error system of follower UAV i at time t, and the symbolic function b i Indicates whether the follower drone can receive the status information of the leader drone, b i =1 means the follower drone can receive information from the leader drone, b i =0 means that the follower drone cannot receive information from the leader drone; a ij =1 indicates that follower UAV i can receive the status information of follower UAV j and they are in a cooperative relationship. ij = -1 indicates that the follower UAV i can receive the status information of the follower UAV j and they are in a competitive relationship. ij = 0 means that follower UAV i cannot receive the status information of follower UAV j; N i represents the set of neighbor follower drones of follower drone i;

[0024] The consumption performance function is expressed as:

[0025]

[0026] in, The consumption performance function of follower drone i at time t, represents the utility function, Q i represents the weight matrix of follower drone i, Q i ≥0; R i represents the symmetric matrix of follower drone i, R i >0.

[0027] Optionally, in step S4, the Critic network evaluates the approximate control action of the Actor network, including the Critic network evaluating the quality of the UAV control action through the output value of the performance function, using the network approximate structure to obtain the approximate performance function, using the Bellman equation to obtain the Bellman performance function, using the gradient descent method to train and adjust the neural network parameters of the Critic network, and in the process of implementing the algorithm, using an event-based trigger mechanism to update the Actor and Critic network parameters.

[0028] Optionally, in step S4, the neural network parameter update formula of the Critic network is expressed as:

[0029]

[0030] in, Indicates that the follower drone i in the Critic network updates the weight parameter at the trigger time, α ci represents the learning rate of follower drone i in the Critic network,

[0031] Optionally, in step S4, the Actor network uses the local state error system information including the follower UAV itself and its neighbor UAVs to approximate the control action; calculates the difference loss between the approximate performance function and the desired final consumption target, and uses the gradient descent method to train and adjust the neural network parameters of the Actor network.

[0032] Optionally, in step S4, the neural network parameter update formula of the Actor network is expressed as:

[0033]

[0034] in, Indicates that the follower drone i in the Critic network updates the weight parameter at the trigger time, α ai represents the learning rate of follower drone i in the Actor network,

[0035] Beneficial effects of the present invention:

[0036] 1. The present invention proposes a new method for designing an event trigger mechanism. Specifically, the present invention selects valid historical data according to the priority in the priority experience playback, and designs event trigger conditions based on this. Compared with the existing event trigger mechanism, the method in the present invention makes full use of the historical information obtained when the drone system interacts with the environment, which not only helps to improve data utilization and save communication resources, but also improves the response speed and stability of the system.

[0037] 2. Based on the designed event trigger mechanism, the present invention proposes a new local error estimation and performance function.

[0038] 3. The unmanned cluster system of the present invention is a system whose precise model is unknown. Compared with the traditional optimization method, the precise model is known and the communication is continuous. However, in many practical situations, the precise model of the system is unknown and continuous communication causes a waste of resources. The Actor-Critic framework and event triggering mechanism used in the present invention can better solve the situation where the precise model of the system is unknown and the communication is continuous.

[0039] 4. The proposed strategy algorithm of fusion event triggering and priority experience replay can make the UAV fully interact with the environment while avoiding continuous communication. The simulation results have been verified. The proposed strategy method of fusion event triggering and priority experience replay can effectively save communication resources and balance system performance and convergence rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0041] Figure 1 It is the overall flow chart of the present invention;

[0042] Figure 2 is a topological diagram that may appear during the system convergence process of the present invention;

[0043] Figure 3 This is a diagram of the evolution of the drone state trajectory of the comparative experiment of the present invention;

[0044] Figure 4 This is a graph of event triggering times of the comparative experiment of the present invention;

[0045] Figure 5 It is the approximate optimization control evolution diagram of the comparative experiment of the present invention;

[0046] Figure 6 It is the state trajectory evolution diagram of the UAV of the present invention;

[0047] Figure 7 It is a UAV event triggering frequency graph of the present invention;

[0048] Figure 8 This is the approximate optimization control evolution diagram of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] like Figure 1 As shown, this embodiment provides a continuous-time unmanned cluster system anti-synchronization optimization control method, which includes but is not limited to the following steps:

[0051] S1: According to the connection status between each drone in the unmanned cluster system, the topological structure and Laplace connection matrix of the unmanned cluster system are constructed; in the embodiment of the present invention, each drone in the unmanned cluster system is connected in a certain way. Figure 2 The following is an example topology diagram of an unmanned swarm system in a leader-follower mode consisting of 8 nodes, where nodes F1, F2, F3, F4, F5, F6, and F7 represent follower drones, and node L1 represents the leader drone. Follower drones F1, F3, and F6 form a group, and the group is a cooperative relationship. F2, F4, F5, and F7 form a group, and the group is a cooperative relationship. F1 and F2 are in a competitive relationship, and F6 and F7 are in a competitive relationship. It is worth noting that the follower drone F1 will obtain information from the leader drone L1.

[0052] Therefore, the relevant parameters of the matrix in the present invention are:

[0053] The topological structure diagram is represented as: G=diag{1,0,0,0,0,0,0};

[0054] The connection matrix is ​​expressed as:

[0055] S2: A policy gradient algorithm that integrates event triggering and priority experience replay mechanism is designed. Based on this, a new local error system is introduced to design a performance indicator function.

[0056] In the embodiment of the present invention, the unmanned swarm system includes a leader and a follower mode, each drone contains its own status information, x i (t) represents the status information of the i-th UAV, x j (t) represents the state information of the jth UAV; through the above method, the leader UAV and the follower UAV are updated using the corresponding leader dynamic equation and follower dynamic equation, which are expressed as:

[0057] The dynamic equation of the leader drone is expressed as:

[0058]

[0059] The dynamic equation of the follower drone is expressed as:

[0060]

[0061] Among them, x0(t) represents the state value of the leader drone at time k, x i (t) represents the status information of the i-th UAV, x j (t) represents the state information of the jth UAV, μ i (t) represents the control input of follower drone i at time t, A, B i are different unknown constant matrices, which have certain suitable dimensions, but their specific values ​​are unknown.

[0062] In the embodiment of the present invention, in order to obtain the event triggering condition, the event triggering time of agent i can be represented by a set of monotonically non-decreasing sequences: To represent, where t i,k <t i,k+1 (k=1,2,...). Define the current error state as and It represents the weighted sampling error state. The error between the two can be defined as:

[0063]

[0064] in, represents the importance sampling weight in the priority experience replay strategy, where M represents the total number of samples stored in the experience replay array, and the hyperparameter β∈(0,1) is introduced as a tuning parameter. p i =ψ(·) is the priority of the i-th agent transition in the experience buffer, and ψ(·) is the mapping function.

[0065] Therefore, the new local error system equation can be expressed as:

[0066]

[0067] Among them, f i is a function of the ith agent.

[0068] Therefore, the event triggering condition is: when the event triggering condition is met, the local error system is asymptotically stable:

[0069]

[0070] Among them, 0<α<1, λ(R ii ),and λ (Q ii ) represent the matrix R ii and Q ii The maximum and minimum eigenvalues ​​of .

[0071] S3: An event-triggered actor-critic neural network is constructed. The network uses the event-triggered mechanism to determine the update time of the network weights, thereby realizing a non-periodic learning process;

[0072] In the embodiment of the invention, a Critic network is used to approximate the performance function, and an Actor network updates the control action according to the performance function;

[0073] Neural network approximation of the performance function:

[0074] Update the control action according to the performance function:

[0075] in, represents the performance function of follower drone i at trigger time k, represents the activation function, Contains input information is the control input information of the neighboring agents of agent i, represents the neural network parameters of the Critic network, Represents the neural network parameters of the Actor network.

[0076] S4: The Critic network evaluates the approximate control actions of the Actor network, and the Actor network adjusts the control actions based on the evaluation of the Critic network. The entire process is updated using gradient descent. A non-periodic update method is used when training the neural network parameters. When the neural network parameters of the Actor-Critic network structure are no longer updated, the approximate optimal consistent control strategy is obtained.

[0077] In an embodiment of the present invention, the Critic network evaluates the control action approximated by the Actor network, including the Critic network evaluating the quality of the drone control action through the output value of the performance function, using the network approximate structure to obtain the approximate performance function, using the Bellman equation to obtain the Bellman performance function, and using the difference function to obtain the differential performance function between the approximate performance function and the Bellman performance function; minimizing the differential performance function, and using the gradient descent method to train and adjust the neural network parameters of the Critic network.

[0078] Among them, for the Critic network:

[0079] The Critic network evaluates the quality of the drone's control actions through the output value of the performance function. The performance function is approximated by the following network structure:

[0080]

[0081] in, represents the performance function of follower drone i at trigger time k, represents the activation function, Contains input information is the control input information of the neighboring agents of agent i, Represents the neural network parameters of the Critic network.

[0082] The neural network parameter update formula of the Critic network is expressed as:

[0083]

[0084] in, Indicates that the follower drone i in the Critic network updates the weight parameter at the trigger time, α ci represents the learning rate of follower drone i in the Critic network,

[0085] In the embodiment of the present invention, the Actor network uses the local state error system information including the follower drone itself and its neighboring drones to approximate the control action;

[0086] Among them, for the Actor network:

[0087] Actor networks are used to approximate control strategies, which are expressed as follows:

[0088]

[0089] in, Represents the neural network parameters of the Actor network.

[0090] The neural network parameter update formula of the Actor network is expressed as:

[0091]

[0092] in, Indicates that the follower drone i in the Critic network updates the weight parameter at the trigger time, α ai represents the learning rate of follower drone i in the Actor network,

[0093] In the embodiment of the present invention, for the control strategy update process in the learning algorithm, hereinafter referred to as Algorithm 1, the present invention adopts the following method to process it, and the specific update method is as follows:

[0094] Step 1: Calculation accuracy ε; iteration index l; admissible control strategy

[0095] Step 2: Strategy update: Consider the control strategy at N event triggering moments and

[0096] Step 3: Strategy improvement: The strategy is improved by the following formula

[0097] Step 4: If the norm of the difference between the two updated Q values ​​is less than a given smaller parameter ε, that is, the formula If ε holds, the iteration is terminated. Otherwise, let l+l+1 and repeat steps 2 to 3.

[0098] In the embodiment of the present invention, for the policy iteration algorithm based on event triggering and priority experience playback, the present invention adopts the following method to train it, hereinafter referred to as Algorithm 2, including the following steps:

[0099] Step 11: Initialization: For any agent i, set the experience pool D M =M capacity is 200, iteration index l = 0;

[0100] Step 22: Event triggering will cause error Store into experience buffer D M ;

[0101] Step 33: For the i-th stored element, when i>M, delete the element stored at the beginning;

[0102] Step 44: Data From the experience buffer D M In the probability P(i) and according to the error Use importance sampling factor α i weighting;

[0103] Step 55: Update Actor network parameters:

[0104]

[0105] Step 66: Update the Critic network parameters:

[0106]

[0107] Step 77: Update priority;

[0108] Step 88: When the event trigger does not occur, the neural network parameters are not updated;

[0109] Step 99: set l=l+1, and return to step 22;

[0110] In order to verify the effectiveness of the proposed strategy iteration algorithm based on event triggering and priority experience playback, Matlab is used for simulation verification. Figure 2 The experimental topology diagram is an unmanned swarm system in a leader-follower mode consisting of 8 nodes, where nodes F1, F2, F3, F4, F5, F6, and F7 represent follower drones, and node L1 represents the leader drone. Follower drones F1, F3, and F6 form a group, and the group is a cooperative relationship. F2, F4, F5, and F7 form a group, and the group is a cooperative relationship. F1 and F2 are in a competitive relationship, and F6 and F7 are in a competitive relationship. It is worth noting that the follower drone F1 will obtain information from the leader drone L1. The relevant parameters in the present invention are:

[0111] Therefore, the relevant parameters of the matrix in the present invention are:

[0112] The topological structure diagram is represented as: G=diag{1,0,0,0,0,0,0};

[0113] The connection matrix is ​​expressed as:

[0114] The system related parameters are:

[0115] B1=[0.017 0.36] T , B2=[0.025 0.29] T , B3=[0.0420.32] T , B4=[0.027 0.41] T , B5=[0.033 0.40] T , B6 = [0.021 0.30] T , B7=[0.032 0.40] T .

[0116] The learning rate is α ai =α ci =0.01, β = 0.5, the experience pool capacity is 200, and the mapping function ψ(p i )=|p i |.

[0117] According to the simulation results, Figure 6The state trajectory evolution process of the unmanned cluster system is clearly shown. It can be inferred that the unmanned cluster system finally successfully achieved group consistency. In addition, the dynamic evolution process of the approximate optimization controller is shown in Figure 8 As shown in Figure 2, the behavior characteristics of the controller are described in detail. Figure 7 The distribution diagram of the event triggering times of each UAV under the framework of the present invention is presented, which intuitively demonstrates the performance of the algorithm in reducing event triggering.

[0118] In order to further verify the advantages of the present invention, the same UAV dynamic system, topological structure, initial value of system state, Critic weight, Actor weight and other related parameters as those in the comparative experiment can be used.

[0119] Under the same parameter settings, Figure 3 and Figure 6 The results show that compared with Figure 3 , Figure 6 The state convergence speed in is significantly faster, which shows that the algorithm proposed in this invention can significantly improve the convergence efficiency of the unmanned cluster system. In order to further verify and explain the advantages of the proposed algorithm, Figure 5 and Figure 8 The comparative simulation results are shown respectively, and the dynamic evolution process of the controller is described in detail. Through comparative analysis, it can be clearly observed that Figure 8 The control strategy adopted in the proposed method has a faster convergence speed.

[0120] Under the same parameter configuration, Figure 4 and Figure 7 The results show that compared with the number of drone event triggers in Figure 4 The frequency of event triggering shown in Figure 7 The number of drone event triggers in the game has been significantly reduced.

[0121] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which may include: ROM, RAM, disk or CD, etc.

[0122] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A continuous-time unmanned cluster system anti-synchronization optimization control method, characterized in that: The following steps are involved: S1: constructing the topological structure and Laplace connection matrix of the unmanned swarm system according to the connection status and cooperation and competition status between the drones in the unmanned swarm system; S2: Design a policy gradient algorithm that integrates event triggering and priority experience replay mechanism; based on this, introduce a new local error system and then design a performance indicator function; S3: Construct an event-triggered actor-critic neural network; the network uses the event-triggered mechanism to determine the update time of the network weights, thereby realizing a non-periodic learning process; S4: The Critic network evaluates the approximate control actions of the Actor network, and the Actor network adjusts the control actions based on the evaluation of the Critic network. The entire process is updated using gradient descent. A non-periodic update method is used when training the neural network parameters. When the neural network parameters of the Actor-Critic network structure are no longer updated, the approximate optimal consistent control strategy is obtained.

2. The method for anti-synchronization optimization control of a continuous-time unmanned cluster system according to claim 1, characterized in that: The local error system triggered by the event in step S2 is as follows: in, represents the triggering time of drone i event, and t i,k <t i,k+1 , (k=1,2,...), define the current error state as and It represents the weighted sampling error state, α i Represents the importance sampling weight in the priority experience replay strategy.

3. The method for anti-synchronization optimization control of a continuous-time unmanned cluster system according to claim 2, characterized in that: The leader UAV and the follower UAV in step S2 are updated using the corresponding leader dynamic equation and follower dynamic equation; The local state error system of the UAV is reconstructed according to the leader dynamic equation and the follower dynamic equation, and the performance function is determined based on the local state error system of the UAV, where: The dynamic equation of the leader drone is expressed as: Where x0(t)∈R n Indicates the position status of the leader drone; The dynamic equation of the follower drone is expressed as: Among them, x i (t)∈R n and Represent the position state and control input of follower UAV i, A∈R n×n and Represent the dynamic matrix and feedback gain matrix of the system respectively.

4. The method for anti-synchronization optimization control of a continuous-time unmanned cluster system according to claim 3, characterized in that: The local tracking error of the drone is defined as: in, represents the local state error system of follower UAV i at time t, and the symbolic function b i Indicates whether the follower drone can receive the status information of the leader drone; a ij =1 indicates that follower UAV i can receive the status information of follower UAV j and they are in a cooperative relationship; N i represents the set of neighbor follower drones of follower drone i; The consumption performance function is expressed as: in, The consumption performance function of follower drone i at time t, represents the utility function, Q i represents the weight matrix of follower drone i, Q i ≥0; R i represents the symmetric matrix of follower drone i, R i >0.

5. The method for anti-synchronization optimization control of a continuous-time unmanned cluster system according to claim 1, characterized in that: In step S4, the Critic network evaluates the approximate control action of the Actor network, including the Critic network evaluating the quality of the UAV control action through the output value of the performance function, using the network approximate structure to obtain the approximate performance function, using the Bellman equation to obtain the Bellman performance function, and using the gradient descent method to train and adjust the neural network parameters of the Critic network. In the process of implementing the algorithm, the Actor and Critic network parameters are updated using an event-based trigger mechanism.

6. The method for anti-synchronization optimization control of a continuous-time unmanned cluster system according to claim 5, characterized in that: In step S4, the neural network parameter update formula of the Critic network is expressed as: in, Indicates that the follower drone i in the Critic network updates the weight parameter at the trigger time, α ci represents the learning rate of follower drone i in the Critic network, 7. The method for anti-synchronization optimization control of a continuous-time unmanned cluster system according to claim 6, characterized in that: In step S4, the Actor network uses the local state error system information including the follower drone itself and its neighbor drones to approximate the control action; calculates the difference loss between the approximate performance function and the expected final consumption target, and uses the gradient descent method to train and adjust the neural network parameters of the Actor network.

8. The method for anti-synchronization optimization control of a continuous-time unmanned cluster system according to claim 7, characterized in that: In step S4, the neural network parameter update formula of the Actor network is expressed as: in, Indicates that the follower drone i in the Critic network updates the weight parameter at the trigger time, α ai represents the learning rate of follower drone i in the Actor network,

Images