Event-triggered reinforcement learning multi-agent consensus control method and system on time scale

Abstract

This invention proposes a time-scaled event-triggered reinforcement learning multi-agent consensus control method and system, belonging to the field of multi-agent cooperative control technology. A dynamic model of a multi-agent system including a leader and multiple followers is established on a timescale. Based on the state information of the leader and followers, dynamic equations for tracking error and consensus error among agents are constructed. A neural network-based reinforcement learning controller is built to approximate the optimal control strategy and performance index function. An event-triggered mechanism is designed to trigger controller updates only when the agent's local error exceeds a dynamic threshold. By constructing a Lyapunov function on the timescale and solving linear matrix inequalities, bounded consensus is ensured and the Zeno phenomenon is eliminated. This invention achieves unified modeling and distributed secure control of time-scaled multi-agent systems under non-ideal environments, saving communication resources, improving convergence speed, and enhancing system robustness and practicality.

Description

Technical Field

[0001] This invention relates to the field of multi-agent cooperative control technology, and in particular to a time-scaled event-triggered reinforcement learning multi-agent consistency control method and system. Background Technology

[0002] As an arbitrary non-empty closed subset of the real number set, time-scaled dynamics provides core theoretical support for the unified analysis of continuous and discrete dynamic systems. Time-scaled dynamical systems theory breaks down the research barriers between differential and difference equations, adapting to both purely continuous and purely discrete scenarios, and accurately characterizing hybrid dynamic processes exhibiting characteristics of both types of change, thus laying the mathematical foundation for the control research of complex multi-agent systems.

[0003] Cooperative control of multi-agent systems has been widely applied in fields such as swarm robots, unmanned surface vessel (USV) formations, smart grids, and intelligent transportation, with consensus control being a core research direction. However, in real-world scenarios, systems are often affected by non-ideal factors such as network attacks and limited communication bandwidth. For example, in unmanned swarm cooperative tasks, the formation may fall into a mixed state of alternating communication interruptions and recovery due to attacks. Such situations are neither fully connected nor continuously disconnected, placing higher demands on the adaptability of control strategies.

[0004] Existing technologies are mostly based on ideal environment assumptions, focusing solely on the consistency problems of continuous or discrete systems. They lack holistic modeling and analysis of mixed dynamic scenarios, making them difficult to apply directly to complex real-world conditions. Furthermore, traditional control strategies often rely on continuous or periodic communication, which is susceptible to bandwidth limitations leading to data transmission delays and packet loss, increases system energy consumption, and fails to balance control performance across continuous and discrete time domains within a unified framework. Moreover, under non-ideal communication conditions, existing methods lack efficient signal update and optimization mechanisms, making it difficult to ensure system stability while reducing computational load. Therefore, a consistency control method applicable to mixed scenarios and balancing performance and efficiency is urgently needed. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a time-scaled event-triggered reinforcement learning-based multi-agent consensus control method and system. This method not only studies continuous and discrete problems within a unified framework, providing a novel perspective and effective mathematical tools for hybrid dynamic problems involving both continuous and discrete processes, but also enables distributed security control of time-scaled multi-agent systems under network attacks and communication constraints, improving the system's stability and reliability under such disturbances. Furthermore, compared to traditional model-based control strategies, this method uses neural networks to approximate unknown nonlinear dynamics and optimal control strategies online. Even under conditions of unknown nonlinear dynamics and external disturbances, it can still achieve high-precision consensus control, demonstrating good practicality and robustness in real-world engineering scenarios with limited communication.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a time-stamped event-triggered reinforcement learning multi-agent consensus control method, comprising:

[0008] Establish a dynamic model of a multi-agent system with one leader and multiple followers on a time scale;

[0009] Based on the state information of the leader and followers, a tracking error equation and a consistency error dynamic equation are constructed among the agents.

[0010] Based on the equation, a neural network-based reinforcement learning controller is constructed to approximate the optimal control strategy and performance index function;

[0011] Design an event triggering mechanism that triggers the controller update only when the agent’s local error exceeds a dynamic threshold; otherwise, maintain the control signal from the previous moment.

[0012] By constructing Lyapunov functions on timescales and solving linear matrix inequalities, the system is guaranteed to achieve bounded consistency and the Zeno phenomenon is eliminated.

[0013] Secondly, the present invention provides a time-stamped event-triggered reinforcement learning multi-agent consensus control system, comprising:

[0014] The time-scaled modeling module is configured to build a dynamic model of a multi-agent system containing one leader and multiple followers on a time scale.

[0015] The equation construction module is configured to construct tracking error equations and consistency error dynamic equations between the agents based on the state information of the leader and followers;

[0016] The control module is configured to construct a neural network-based reinforcement learning controller based on the equations to approximate the optimal control strategy and performance index function.

[0017] The event triggering module is configured to design an event triggering mechanism that triggers the controller update only when the agent’s local error exceeds a dynamic threshold; otherwise, the control signal from the previous moment is maintained.

[0018] The solver module is configured to ensure the system achieves bounded consistency and eliminates the Zeno phenomenon by constructing Lyapunov functions on timescales and solving linear matrix inequalities.

[0019] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the time-stamped event-triggered reinforcement learning multi-agent consensus control method described in the first aspect.

[0020] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the time-scaled event-triggered reinforcement learning multi-agent consistency control method described in the first aspect.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0022] (1) Time-scaled modeling. This invention proposes to establish system models and control strategies within a unified framework of "time scales." The proposed time-scaled model uniformly describes continuous, discrete, and hybrid systems, providing a theoretically rigorous and engineering-practical solution for multi-agent cooperative control under non-ideal environments such as intermittent faults, communication bandwidth variations, or network attacks. This technology can be widely applied in key areas such as UAV swarm cooperative operations, unmanned surface vessel formations, smart grid scheduling, and emergency communication networks, improving the robustness of the system under non-ideal environments and supporting the reliable operation of national defense security and critical infrastructure.

[0023] (2) Time-scaled event-triggered mechanism. This mechanism avoids prior assumptions about the continuity or discreteness of communication modes, triggering controller updates only when the agent's local error exceeds a dynamic threshold. This mechanism effectively reduces the frequency of control signal updates and the number of agent event triggers, thereby reducing network bandwidth usage and node energy consumption. While saving communication resources, it effectively improves the system convergence speed, solving the problem that traditional control often struggles to balance these two aspects. It provides a feasible solution for deploying high-performance cooperative control algorithms in real-world scenarios with limited communication, such as distributed sensor networks and large-scale UAV swarms.

[0024] (3) Neural Network Reinforcement Learning. This invention introduces a reinforcement learning mechanism based on the Actor-Critic network for the first time in a time-scaled multi-agent system, enabling the system to autonomously learn and optimize control strategies under resource-constrained conditions, thereby improving control performance while saving communication and computing resources. By approximating unknown nonlinear dynamics and optimal control strategies online through neural networks, high-precision consistent control can still be achieved even when the nonlinear dynamics are unknown and external disturbances exist. This technology provides a feasible solution to common engineering challenges in high-end equipment, intelligent robots, and the Industrial Internet of Things, such as poor real-time performance, limited onboard computing power, and high communication load.

[0025] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0027] Figure 1 This is a flowchart of the main process of a time-stamped event-triggered reinforcement learning multi-agent consensus control method provided in an embodiment of the present invention;

[0028] Figure 2 A flowchart of a time-stamped event-triggered reinforcement learning multi-agent consensus control method provided in an embodiment of the present invention;

[0029] Figure 3 Time scale provided for embodiments of the present invention A schematic diagram illustrating the position tracking error of a leader-follower agent.

[0030] Figure 4 Time scale provided for embodiments of the present invention A schematic diagram illustrating the velocity tracking error of a leader-follower agent.

[0031] Figure 5 Time scale provided for embodiments of the present invention A diagram illustrating the event triggering moments of the follower agent;

[0032] Figure 6 Time scale provided for embodiments of the present invention A schematic diagram of the control inputs of the follower agent;

[0033] Figure 7 Time scale provided for embodiments of the present invention A schematic diagram illustrating the position tracking error of a leader-follower agent.

[0034] Figure 8 Time scale provided for embodiments of the present invention A schematic diagram illustrating the velocity tracking error of a leader-follower agent.

[0035] Figure 9 Time scale provided for embodiments of the present invention A diagram illustrating the event triggering moments of the follower agent;

[0036] Figure 10 Time scale provided for embodiments of the present invention A schematic diagram of the control inputs of the follower agent;

[0037] Figure 11 A schematic diagram of the position tracking error of a leader-follower agent on a hybrid timescale provided in an embodiment of the present invention;

[0038] Figure 12 A schematic diagram of the velocity tracking error of a leader-follower agent on a hybrid timescale provided in an embodiment of the present invention;

[0039] Figure 13 This is a schematic diagram of the event triggering time of a follower agent on a hybrid timescale provided in an embodiment of the present invention;

[0040] Figure 14 This is a schematic diagram of the control input for a follower agent on a hybrid timescale provided in an embodiment of the present invention;

[0041] Figure 15 This is a schematic diagram illustrating the position tracking error of a leader-follower agent on a hybrid time scale under a parameter self-optimization algorithm provided in this embodiment of the invention.

[0042] Figure 16 This is a schematic diagram illustrating the velocity tracking error of a leader-follower agent on a hybrid time scale under a parameter self-optimization algorithm provided in an embodiment of the present invention.

[0043] Figure 17 This is a schematic diagram illustrating the event triggering times of a follower agent on a hybrid timescale under the parameter self-optimization algorithm provided in this embodiment of the invention.

[0044] Figure 18 This is a schematic diagram of the control input of a follower agent on a hybrid timescale under the parameter self-optimization algorithm provided in an embodiment of the present invention;

[0045] Figure 19 This is a parameter setting interface for a multi-agent system parameter optimization GUI provided in an embodiment of the present invention;

[0046] Figure 20 This is the optimization result interface of the GUI for optimizing parameters of a multi-agent system provided in this embodiment of the invention;

[0047] Figure 21 This is a visualization interface for the results of the GUI for optimizing parameters of a multi-agent system provided in an embodiment of the present invention. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] Explanation of technical terms

[0050] 1. Time scale: Time scale It is the set of real numbers For any non-empty closed subset, common timescales include:

[0051] (1) Continuous time, Time is continuous and uninterrupted;

[0052] (2) Discrete time, Time is a discrete point with a fixed interval; For time intervals, , Represents the set of integers;

[0053] (3) Hybrid time, where time is a mixture of "continuous segments" and "discrete intervals", such as .

[0054] 2. Forward jump operator: For any Define the pre-jump operator for: s indicates time scale Any element in the timescale; the forward jump operator is used to determine the first time after time t in the timescale.

[0055] 3. Distance Function: Define the distance function. for: That is, the forward jump operator With the current moment The difference is used to characterize the discreteness or continuity of the time scale. As a unified metric, it enables control algorithms to adapt to different time scales without requiring separate algorithms to be designed for continuous or discrete systems.

[0056] 4. Time-scaled derivative: Let the function... If it exists , such that for any ,exist a neighborhood (i.e., for a certain) ,have ), satisfying all :

[0057] ;

[0058] Then it is called for At point place -Derivatives. Time-scaled derivatives unify and generalize continuous differentials and discrete differences, allowing the description of system dynamics at different time scales, including continuous, discrete, and hybrid systems, within a unified framework.

[0059] Example 1

[0060] like Figure 1As shown, this embodiment discloses a time-stamped event-triggered reinforcement learning multi-agent consensus control method, including the following steps:

[0061] S1: Establish a dynamic model of a multi-agent system with one leader and multiple followers on a time scale;

[0062] S2: Based on the state information of the leader and followers, construct the tracking error equation and the consistency error dynamic equation between the agents;

[0063] S3: Construct a reinforcement learning controller based on a neural network according to the equation to approximate the optimal control strategy and performance index function;

[0064] S4: Design an event triggering mechanism that triggers the controller update only when the agent’s local error exceeds the dynamic threshold; otherwise, maintain the control signal from the previous moment.

[0065] S5: By constructing Lyapunov functions on the timescale and solving linear matrix inequalities, the system is guaranteed to achieve bounded consistency and the Zeno phenomenon is eliminated.

[0066] Next, combined Figure 2 This embodiment provides a detailed description of a time-stamped event-triggered reinforcement learning multi-agent consistency control method.

[0067] (I) System Modeling and Initialization

[0068] First, construct a multi-agent system. (At the time scale) Establish a system that includes a leader (labeled as agent 0) and Followers (marked as number 1 to number 1) A second-order nonlinear multi-agent system dynamics model (agent number 1).

[0069] In time scale The dynamic model for each agent is defined as follows:

[0070] Leader Dynamic Model:

[0071]

[0072] Follower dynamics model:

[0073]

[0074] Among them, time scale Let be any non-empty closed subset of the set of real numbers, which can represent continuous time (e.g. ), discrete time (e.g.) or mixed time domain; Representing intelligent agents respectively Position and velocity states, , These represent the leader's position and speed, respectively. This refers to the derivative operation in time-scaled calculus, used to describe the rate of change of a state at a given time scale; This is the control input to be designed, where m represents the dimension of the state space. and These are descriptions of the leader system and the... The nonlinear bounded function of the intrinsic dynamics of a follower system.

[0075] Next, the communication topology is defined. (Figure) For a multi-agent topology, where the graph... It is an undirected graph, or a strongly connected directed graph, or it contains a directed spanning tree.

[0076] A collection of intelligent agents, For edge set, It is an adjacency matrix. For the first The first agent and the second The weight of communication connections between agents. Indicates the first The agent can start from the first An intelligent agent acquires information. If... ,but ;otherwise, .

[0077] The degree matrix is ​​defined as ,in . The Laplacian matrix is ​​defined as The adjacency matrix of the leader It represents the connection between leader and follower intelligent agents. Indicates the first The connection weights between each follower agent and the leader. If the first... If an agent is adjacent to the leader agent, then ,otherwise .

[0078] Further, initialize system parameters:

[0079] 1. Set time scale ,For example , Or a mixed time domain;

[0080] 2. Set event trigger parameters and ; This is the proportional threshold parameter. The threshold parameter is a constant.

[0081] 3. Set reinforcement learning parameters, including the Actor network learning rate. Critic network learning rate and Identifier network update rate Neural network parameters, including the number of hidden layer nodes. Radial basis function (RBF) center and width ;

[0082] 4. Set the initial position and speed of the leader and follower agents.

[0083] (II) Controller Design

[0084] To handle unknown nonlinear dynamics in multi-agent systems and achieve adaptive optimal control, this embodiment constructs a reinforcement learning controller based on neural networks.

[0085] This controller employs an Actor-Critic framework and works in conjunction with an Identifier network to form an adaptive control unit. The core of the controller lies in approximating the gradient of the optimal performance metric function and the optimal control strategy through neural networks, with all neural network weight updates performed only at the trigger moment. The controller includes the following core neural network modules:

[0086] (1) Identifier network: used to approximate the unknown nonlinear dynamic error between follower and leader agents, defined as:

[0087]

[0088] The output of this network is:

[0089]

[0090] in, It is the weight matrix of the Identifier network. It is a Gaussian radial basis function vector. This represents the number of hidden layer nodes. This is the network input vector. This network provides system dynamics information for subsequent controller design. The weight update rule is:

[0091]

[0092] in, This is the learning rate.

[0093] (2) Critic network: used to estimate the gradient of the optimal performance index function. Its output is:

[0094]

[0095] in, Let it be a distance function. Represents intelligent agents The estimated optimal performance index function is used to approximate the true performance index. That is, the minimum cumulative cost starting from the current state. Represents the integral variable. For position consistency error, For speed consistency error, Indicates the first Consensus error vector of each agent , , These represent position tracking gain and velocity tracking gain, respectively. Here is the weight matrix of the Identifier network. Here is the weight matrix of the Critic network. For the first The RBF basis function vectors of the Identifier network for each agent. For the first The RBF basis function vectors of the Critic network for each agent. For the first The neighbor weights of each agent.

[0096] The Critic network weights are updated only at the trigger time:

[0097]

[0098] in, This represents the weight matrix of the Critic neural network. Represents the integral variable. is the learning rate of the Critic network.

[0099] (3) Actor Network: Generates an approximate optimal control strategy based on the output of the Critic Network. Its output is the control law:

[0100]

[0101] in, Here is the weight matrix of the Actor network. represents the RBF basis function vector of the Actor network.

[0102] The Actor network weights are updated at the trigger time:

[0103]

[0104] in, is the learning rate of the Actor network.

[0105] (4) Network collaboration and updates

[0106] Based on the event-triggered mechanism on the time scale, the three networks mentioned above can work together:

[0107] (a) When the triggering condition is met, the network weights of Identifier, Critic, and Actor are updated synchronously;

[0108] (b) At non-triggered times, each network weight retains the value of the previous trigger time, and the controller output also remains unchanged.

[0109] This mechanism significantly reduces the communication and computational burden while ensuring the system's bounded consistency.

[0110] Furthermore, to conserve communication resources, this embodiment employs event-triggered control on a time scale. Control input is only applied when the following conditions are met. Update:

[0111]

[0112] in, :

[0113]

[0114] in, Indicates transpose. , Indicates different followers; For intelligent agents The set of neighbors. , They represent the first The position and velocity states of each follower; , They represent the first The position and velocity states of each follower.

[0115] The triggering condition (12) is based on the agent's own tracking error and the state information of neighboring agents. The event is triggered only when the calculated error function exceeds a state-related threshold. During non-triggering times... The control signal and neural network weights remain unchanged from the results of the previous triggering time. This reduces the communication and computing burden.

[0116] (III) Online Learning and Control Execution

[0117] At each sampling time, the agent acquires its own state and the states of its neighbors, calculates the tracking error, and determines whether the current event trigger condition is met. If triggered, the neural network weights are updated and a new control input is calculated; otherwise, the control input and network weights from the previous trigger time remain unchanged.

[0118] The process is as follows:

[0119] 1. Start: System initialization, parameter setting, and neural network weight initialization.

[0120] 2. Loop: For each time step ,

[0121] (1) State awareness: Each follower Measure its own state It also obtains the state information of neighboring intelligent agents and the state information of the leader through the communication network.

[0122] (2) Error Calculation: Calculate the tracking error and consistency error (13).

[0123] (3) Event trigger judgment: Check whether the trigger conditions are met.

[0124] (a) If triggered:

[0125] Update Identifier: Update the network weights of Identifier according to the weight update law (5).

[0126] Update the Critic network: Update the weights of the Critic network according to the weight update laws (7)-(8). ;

[0127] Update the Actor network: Update the Actor network weights according to the weight update laws (10)-(11). ;

[0128] Calculate the control input: using the latest Actor network weights, according to the control law:

[0129]

[0130] Calculate the current control input .

[0131] (b) If not triggered:

[0132] The neural network weights retain their values ​​from the previous trigger moment, and the control input remains unchanged. .

[0133] (4) Control inputs act on the system to drive system state updates.

[0134] 3. Termination judgment: Continue running until the system reaches and maintains a leader-follower bounded consistency state.

[0135] (iv) Stability verification

[0136] By constructing a Lyapunov function and proving that bounded consistency can be achieved for second-order multi-agents on the timescale under the condition of satisfying the linear matrix inequality (LMI) proposed in this embodiment, the Zeno phenomenon is excluded.

[0137] The Lyapunov function is as follows:

[0138]

[0139] Among them, the system tracking error term Used to evaluate the overall tracking error of the system. It is a positive definite symmetric matrix; This represents the position and velocity tracking error of all agents. ,in For position tracking error, For speed tracking error, express The identity matrix;

[0140] Identifier network weight estimation error term Used to evaluate the weight error of the Identifier network. ;in, Indicates the first The ideal weight matrix of the Identifier network for each agent; Represents the trace of a matrix;

[0141] Critic network weight estimation error term Used to evaluate the weight error of the Critic network. ; in, Indicates the first The ideal weight matrix for an Actor or Critic network of agents.

[0142] Actor network weight estimation error term Used to evaluate the weight error of the Actor network. ;

[0143] And prove that if positive constants exist and positive definite symmetric matrix If the linear matrix inequality (16) holds, then the second-order multi-agent system on the timescale can achieve bounded consistency and eliminate the Zeno phenomenon. The specific form of the linear matrix inequality (LMI) is:

[0144]

[0145]

[0146]

[0147]

[0148]

[0149]

[0150]

[0151]

[0152]

[0153]

[0154] ,

[0155] in, , , , ,

[0156] , They are peacekeeping 3D identity matrix , They are peacekeeping Zero-dimensional matrix For a graph Laplacian matrix with a leader, , The largest eigenvalue of the matrix. It represents the Kronecker product.

[0157] To verify the effectiveness of this embodiment, the following specific embodiment is provided:

[0158] Example 1. Unmanned Aerial Vehicle (UAV) Formation Assembly

[0159] This embodiment simulates a drone swarm formation task. In this scenario, the drone swarm needs to quickly complete formation under relatively good communication conditions and high requirements for formation speed. The system uses a small communication interval to achieve high control accuracy and fast convergence speed when communication bandwidth is sufficient.

[0160] Consider a swarm consisting of one lead drone and eight follower drones, with a directed graph as its communication topology. The positions and velocities of all follower drones gradually approach those of the lead drone, eventually maintaining a bounded error. The nonlinear dynamic function of the system is:

[0161]

[0162] Control parameters , Event triggering parameters , Learning rate , Neural network weights , The Critic and Actor networks each have 10 nodes. The RBF network's center... exist arrive Evenly distributed between them, width The leader's initial position and speed are... The follower's initial position and speed are ( ).

[0163] make It can be verified that under the above parameter conditions, the linear matrix inequality condition (16) holds. Figure 3 and Figure 4 The time stamp is shown in the event triggering mechanism designed in this embodiment. In a multi-agent system, position and velocity tracking errors between the leader and followers are addressed, and the system achieves bounded consistency. Figure 5 This demonstrates the event triggering moments of the follower agent. Figure 6 For the corresponding control input. In Figure 3 , Figure 4 As should be understood in similar figures below, in this type of simulation study, to facilitate a visual demonstration of the control effect, state variables such as position and velocity are rendered dimensionless. Therefore, the vertical axis is a dimensionless value and there is no need to label it with specific physical units.

[0164] Based on practical application requirements, the time required to achieve bounded consistency can be determined by satisfying condition (18). To measure: a tracking error threshold for a given location and speed tracking error threshold :

[0165]

[0166] If we take in equation (18) and Then in Within this period, the total number of event triggers in the system was 1918, and in equation (18) It takes 5.20 seconds.

[0167] Example 2. Communication resources are limited by the aggregation of drone swarms.

[0168] This embodiment simulates a drone swarm assembly task where communication resources are limited. The system uses a larger communication step size, making... .

[0169] The topology and other parameters between the leader and follower agents are the same as in Example 1, and it can be verified that condition (16) holds under the above parameters. Figure 7 and Figure 8 Specifically, under the event-triggered controller designed in this invention, the time stamp... Position and velocity tracking errors between the leader and follower agents in a multi-agent system. Figure 9 It is the event trigger moment for the follower agent. Figure 10 It is the control input. If we take in equation (18) and Then in Within this period, the total number of system events triggered was 2300, and in equation (18) It takes 6.28 seconds.

[0170] Example 3. Dynamic Step Size Adaptive Hybrid Control Scenario

[0171] This embodiment simulates a collaborative assembly task of an adaptive UAV swarm in a dynamic and complex environment. This scenario requires the system to autonomously adjust its communication and control strategies based on real-time environmental changes, balancing convergence speed and communication resource consumption. Specifically, a dynamic step-size switching strategy is introduced, where the step size is dynamically adjusted based on the system's convergence status and error magnitude.

[0172] In the early stages of convergence, if the maximum error exceeds the threshold of 1.5, a larger step size is used. To achieve fast convergence. Otherwise, use a smaller step size ( We will implement precise control to ensure that the final assembly accuracy meets the mission requirements. Figure 11 and Figure 12 These refer to the position and velocity tracking errors between the leader and follower agents in a hybrid temporal multi-agent system under the event-triggered controller designed in this embodiment. Figure 13 It is the event trigger moment for the follower agent. Figure 14 It's a control input. Within the range The number of events triggered within the timeframe is 1827, and in equation (18) The event trigger time is 5.05 seconds. (This refers to the event trigger time in three different scenarios.) The comparison is shown in Table 1, which indicates that control in the mixed time domain is more effective. Compared with Case 1 and Case 2, Case 3 reduced the total number of triggers in the mixed domain by approximately 4.7% and 20.6%, respectively.

[0173] Table 1. Comparison of trigger counts and convergence times between fixed step size and mixed step size;

[0174]

[0175] Table 1 compares the number of event triggers and convergence time for the three scenarios. It can be seen that under the hybrid time-domain control strategy,

[0176] (1) with fixed step size In comparison, the number of event triggers decreased by 4.7%, and the convergence time was shortened by 2.9%.

[0177] (2) with fixed step size In comparison, the number of event triggers decreased by 20.6%, and the convergence time was shortened by 19.6%.

[0178] This embodiment demonstrates that the hybrid time-domain control strategy proposed in this invention can effectively adapt to the dynamic changes in communication status and effectively improve control efficiency in practical applications; at the same time, by adopting a dynamic step size switching strategy, the utilization efficiency of communication resources is optimized while ensuring system stability.

[0179] Example 4. Performance Verification of Optimal Parameters

[0180] To verify the performance advantage of the hybrid time-domain strategy with optimal parameters, this embodiment compares the performance of the fixed time step and the hybrid time-domain strategy in terms of the number of event triggers and system convergence time by performing system optimization on the event trigger parameters. Figure 15 and Figure 16 These are the position and velocity tracking errors of the leader-follower agents on a mixed time scale under the event-triggered controller and parameter self-optimization algorithm designed in this embodiment. Figure 17 It is the event trigger moment for the follower agent. Figure 18 It controls the input.

[0181] Table 2 lists the three strategies under their respective optimal parameters. The number of events triggered and the system convergence time.

[0182] Table 2. Number of event triggers and convergence time for fixed step size and mixed time domain under optimal parameters;

[0183]

[0184] As shown in Table 2, under their respective optimal parameter conditions, the hybrid time-domain control strategy in this embodiment not only reduces the number of event triggers, but also shortens the system convergence time, solving the problem that traditional control methods cannot balance trigger frequency and convergence speed.

[0185] (1) with fixed step size In comparison, the hybrid time-domain strategy further reduced the number of event triggers by 4.7% and shortened the convergence time by 2.9%.

[0186] (2) with fixed step size In comparison, the hybrid time-domain strategy further reduced the number of event triggers by 20.6% and shortened the convergence time by 6.5%.

[0187] This embodiment demonstrates that, even with their respective optimal parameter settings, the hybrid time-domain strategy can still balance reducing the number of event triggers and accelerating system convergence, thereby improving control efficiency while saving communication resources.

[0188] Example 5. Implementation of Parameter Self-Optimization Algorithm

[0189] Through the foregoing embodiments, it can be observed that the hybrid time-domain strategy is more efficient than the fixed-step-size control. To further optimize system performance, this embodiment proposes a parameter self-optimization algorithm, which can automatically search for the optimal parameter combination for a given bounded consistency requirement. Its optimization objective is to minimize the number of event triggers and shorten the convergence time while ensuring stable system convergence.

[0190] For bounded consistency goals

[0191] , , , ;

[0192] The objective function is defined as minimizing the number of event triggers and the convergence time, i.e.:

[0193] ;

[0194] in, This represents the total number of times the event was triggered. To achieve the minimum time required for bounded consistency, Let be the convergence time of a certain baseline strategy. This represents the number of times a certain baseline strategy is triggered. As weight.

[0195] For example, setting the same bounded consistency target as in the aforementioned embodiments is... The optimal parameter combinations can be obtained as shown in Table 3:

[0196] Table 3. Optimal parameter combinations;

[0197]

[0198] Among them, when the position and velocity tracking errors of all leader and follower agents are less than or equal to At this point, the step size switches from 0.03 to 0.005, and the position tracking error of all leader and follower agents is less than or equal to... Speed ​​tracking error less than or equal to At that time, the step size switches from 0.005 to 0.01.

[0199] Under the above parameters, the total number of triggers for the multi-agent system was 1395, and the convergence time was 4.45s.

[0200] (1) with fixed step size In comparison, the hybrid domain strategy combined with the parameter liberalization algorithm further reduced the number of event triggers by 27.3% and shortened the convergence time by 14.4%.

[0201] (2) with fixed step size In comparison, the hybrid domain strategy combined with the parameter liberalization algorithm further reduced the number of event triggers by 39.4% and shortened the convergence time by 17.6%.

[0202] For the same bounded consistency objective, the introduction of a parameter self-optimization algorithm enables the system to achieve faster convergence with fewer event triggers. These advantages make the method valuable for engineering applications in real-world scenarios with limited communication bandwidth, computational power constraints, and complex dynamic characteristics, such as collaborative scheduling of intelligent warehousing robots, drone swarms in emergency communication networks, and intermittently connected industrial IoT.

[0203] Example 6. Implementation of a Graphical User Interface (GUI) for Parameter Optimization in a Multi-Agent System

[0204] To facilitate user operation, this invention provides a graphical user interface for multi-agent system parameter optimization. The modular interface integrates parameter configuration, optimization algorithms, and result visualization, allowing users to focus solely on the consistency objective without needing in-depth knowledge of the underlying theories and algorithmic details. The system automatically invokes a built-in hierarchical optimization algorithm to intelligently search for optimal control parameters, event-triggered parameters, and neural network learning rates within a given parameter space, ultimately outputting the optimal parameter combination that simultaneously reduces communication frequency and accelerates convergence. This design enhances the practicality and engineering applicability of the method.

[0205] This GUI adopts a modular design and includes three main function tabs, enabling a complete workflow from parameter setting and optimization calculation to result visualization. Figures 19-21 As shown.

[0206] (I) GUI Interface Layout and Functional Modules

[0207] The GUI interface uses a three-tab design, specifically including:

[0208] (1) Parameter Settings tab:

[0209] Agent settings panel: Provides input controls for the number of agents, supporting dynamic configuration of 1-20 agents.

[0210] Communication topology settings: Includes two editable matrix input tables, used to configure the communication topology matrix A between follower agents and the connection matrix B between follower and leader agents, respectively.

[0211] System parameter settings: Only consistency targets need to be entered. And the total simulation time T.

[0212] Run Optimization Button: Clicking this button will start the parameter optimization calculation process.

[0213] (2) Optimize the results tab:

[0214] Optimal Parameter Combination Display: Shows the optimal parameters obtained through optimization, including:

[0215] Time step parameters , , Controller parameters , Event triggering parameters , ,

[0216] Neural network parameters , , Switch threshold , , .

[0217] Optimization results statistics: Displays the total number of triggers, convergence time, and whether the optimization was successful.

[0218] (3) Results Visualization Tab:

[0219] The following key performance indicators are displayed graphically using a two-row, three-column layout:

[0220] The first line contains the position tracking curve, velocity tracking curve, and event trigger time distribution.

[0221] The second line controls the input curve, the norm change of the Actor network weights, and the norm change of the Critic network weights.

[0222] (II) Parameter Optimization Algorithm Flow

[0223] 1. Time step parameter optimization

[0224] The test uses convergence time and event trigger count weighted cost as the evaluation metric. , , Different combinations.

[0225] 2. Controller parameter optimization

[0226] (1) Optimize controller parameters based on the optimal time step , .

[0227] (2) Perform an iterative search based on the optimal result of the previous step.

[0228] 3. Optimization of event-triggered parameters and neural network parameters

[0229] (1) Optimize event triggering parameters , and step size switching threshold , , To balance triggering frequency and control precision.

[0230] (2) Collaborative optimization of neural network parameters , , This ensures the convergence and approximation accuracy of the neural network.

[0231] 4. Final Simulation and Result Display

[0232] Run the final simulation with optimal parameters, collect all performance data, and update the GUI display.

[0233] (III) Example of Operation Procedure

[0234] The user's optimized operation process via the GUI is as follows:

[0235] 1. Parameter initialization: The system automatically loads the default parameters when it starts up, and all graphics areas are initialized to a blank state.

[0236] 2. Parameter Adjustment: Users can modify parameters in the parameter settings tab. When the number of agents changes, the dimension of the communication topology matrix will be automatically adjusted synchronously.

[0237] 3. Operation Optimization: Click the "Operation Optimization" button to start the optimization process. The status panel displays the current optimization steps and progress in real time.

[0238] 4. Results Display: After optimization, the optimal parameter combination will be displayed on the Optimization Results tab. The performance comparison chart will be updated and displayed on the Results Visualization tab.

[0239] 5. Results Analysis: Users can intuitively judge the system performance through graphics, and it supports interactive operations such as graphics scaling and panning. It can run optimization repeatedly and compare the control effects under different parameter settings.

[0240] To address the challenge of traditional methods in achieving unified modeling and cooperative control of continuous, discrete, and hybrid systems under non-ideal environments such as network attacks and communication constraints, this invention proposes a distributed cooperative control strategy within a time-scaled framework. This strategy involves establishing a second-order nonlinear multi-agent system dynamic model on the time scale, designing position and velocity tracking errors, and constructing a consistency error dynamic system. An event-triggered mechanism based on an error threshold is designed, updating the control signal only when triggering conditions are met, significantly reducing communication frequency. A reinforcement learning controller based on neural network approximation is constructed, using an Actor-Critic network structure to approximate the optimal performance index function and control strategy, respectively. The neural network weights are updated only at event triggers, and the system achieves semi-global consensus and eventual bounded consistency by solving linear matrix inequalities. This invention effectively overcomes the limitations of traditional cooperative control in non-ideal environments, ensuring control accuracy while reducing communication burden, and significantly improving the robustness and practicality of multi-agent systems in real-world engineering scenarios.

[0241] Example 2

[0242] This embodiment provides a time-stamped event-triggered reinforcement learning multi-agent consensus control system, including:

[0243] The time-scaled modeling module is configured to build a dynamic model of a multi-agent system containing one leader and multiple followers on a time scale.

[0244] The equation construction module is configured to construct tracking error equations and consistency error dynamic equations between the agents based on the state information of the leader and followers;

[0245] The control module is configured to construct a neural network-based reinforcement learning controller based on the equations to approximate the optimal control strategy and performance index function.

[0246] The event triggering module is configured to design an event triggering mechanism that triggers the controller update only when the agent’s local error exceeds a dynamic threshold; otherwise, the control signal from the previous moment is maintained.

[0247] The solver module is configured to ensure the system achieves bounded consistency and eliminates the Zeno phenomenon by constructing Lyapunov functions on timescales and solving linear matrix inequalities.

[0248] Example 3

[0249] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a time-stamped event-triggered reinforcement learning multi-agent consensus control method as described in Embodiment 1 above.

[0250] Example 4

[0251] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the event-triggered reinforcement learning multi-agent consistency control method on time scales as described in Embodiment 1 above.

[0252] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0253] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A time-stamped event-triggered reinforcement learning multi-agent consistency control method, characterized in that, Includes the following steps: Establish a second-order nonlinear multi-agent system dynamics model with one leader and multiple followers on a time scale; Based on the state information of the leader and followers, a tracking error equation and a consistency error dynamic equation are constructed among the agents. The consistency error dynamic equation describes the error evolution through time-scaled derivatives. The consistency error includes position consistency error and velocity consistency error, and incorporates the Laplacian matrix information of the multi-agent communication topology. A reinforcement learning controller based on a neural network is constructed according to the equation to approximate the optimal control strategy and performance index function. The reinforcement learning controller includes an Actor-Critic-Identifier three-network, which is constructed based on an RBF neural network. The Identifier network is used to approximate the unknown nonlinear dynamic errors of the leader and followers, the Critic network is used to estimate the gradient of the optimal performance index function, and the Actor network is used to output an approximate optimal control law. An event-triggered mechanism is designed to trigger the controller update only when the agent's local error exceeds a dynamic threshold; otherwise, the control signal from the previous moment is maintained. The controller adopts a time-scaled event-triggered mechanism, defining a time-scaled sequence composed of event trigger moments. The weights of the three networks are synchronously updated only at each trigger moment in the time-scaled sequence. By constructing a Lyapunov function on the time-scaled sequence and solving linear matrix inequalities, the system achieves bounded consistency and eliminates the Zeno phenomenon. The Lyapunov function consists of a system tracking error term, an Identifier network weight estimation error term, a Critic network weight estimation error term, and an Actor network weight estimation error term.

2. The event-triggered reinforcement learning multi-agent consistency control method on time scales as described in claim 1, characterized in that, The timescale is any non-empty closed subset of the set of real numbers, used to uniformly describe the continuous time domain, discrete time domain, or mixed time domain.

3. The event-triggered reinforcement learning multi-agent consistency control method on time scales as described in claim 1, characterized in that, In the tracking error equation, the position tracking error is defined as the position difference between the follower and the leader, and the velocity tracking error is defined as the velocity difference between the follower and the leader.

4. The event-triggered reinforcement learning multi-agent consistency control method on time scales as described in claim 1, characterized in that, By constructing Lyapunov functions on timescales and solving linear matrix inequalities, the system achieves bounded consistency and eliminates the Zeno phenomenon. Specifically, this includes: By taking the derivative of the Lyapunov function and using the event triggering conditions and neural network update rules, sufficient conditions for system stability are derived, and these conditions are transformed into linear matrix inequalities. By solving the linear matrix inequality, a set of positive definite symmetric matrices and positive constants are obtained, which makes the tracking error and network weight estimation error of the system consistent and bounded under the given conditions, and the event triggering interval has a positive lower bound, thus eliminating the Zeno phenomenon.

5. The event-triggered reinforcement learning multi-agent consistency control method on time scales as described in claim 1, characterized in that, The time scale supports a dynamic step size switching strategy, which adaptively adjusts the distance function according to the real-time convergence status of the system and the magnitude of the tracking error. In the early stages of convergence and when the tracking error exceeds a preset threshold, a step size of the first value is adopted; When the tracking error decreases to below the threshold, the step size is switched to the second value; the first value is greater than the second value.

6. A time-scaled event-triggered reinforcement learning multi-agent consensus control system, characterized in that, include: The time-scaled modeling module is configured to build a second-order nonlinear multi-agent system dynamics model containing one leader and multiple followers on a timescale. The equation construction module is configured to construct tracking error equations and consistency error dynamic equations between the agents based on the state information of the leader and followers; The consistency error dynamic equation describes the error evolution through time-scaled derivatives. The consistency error includes position consistency error and velocity consistency error, and incorporates the Laplacian matrix information of the multi-agent communication topology. The control module is configured to construct a neural network-based reinforcement learning controller according to the equation, approximating the optimal control strategy and performance index function; the reinforcement learning controller includes an Actor-Critic-Identifier three-network, which is constructed based on an RBF neural network; the Identifier network is used to approximate the unknown nonlinear dynamic errors of the leader and followers, the Critic network is used to estimate the gradient of the optimal performance index function, and the Actor network is used to output an approximate optimal control law; The event triggering module is configured to design an event triggering mechanism that triggers the controller update only when the agent's local error exceeds a dynamic threshold; otherwise, the control signal of the previous moment is maintained. The controller adopts a time-stamped event triggering mechanism, defining a time-stamped sequence composed of event triggering moments. The weights of the three networks are synchronously updated only at each triggering moment in the time-stamped sequence. The solver module is configured to ensure that the system achieves bounded consistency and eliminates the Zeno phenomenon by constructing a Lyapunov function on the time scale and solving linear matrix inequalities. The Lyapunov function consists of a system tracking error term, an Identifier network weight estimation error term, a Critic network weight estimation error term, and an Actor network weight estimation error term.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the event-triggered reinforcement learning multi-agent consistency control method on a time scale as described in any one of claims 1-5.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the event-triggered reinforcement learning multi-agent consistency control method on a time scale as described in any one of claims 1-5.

Images