Multi-agent cooperative neural network control method, controller construction method and system
By building a neural network controller in a multi-agent system and using trajectory error to train the neural network weight, the problems of unpredictable system status and high communication bandwidth are solved, resource utilization optimization and communication load reduction are achieved, and the stability and efficiency of the system are improved.
Patent Information
- Application Number
- CN202510539535.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-27
- Publication Date
- 2025-08-12
AI Technical Summary
In multi-agent systems, the prior art is difficult to effectively solve the problems of unpredictable system status and high communication bandwidth requirements, resulting in non-causal problems and waste of resources.
By predicting future state vectors, building a neural network controller, using the trajectory error of the agent to train the neural network weights, design a stable neural network update rate, realize offline controller design, and reduce communication load.
It realizes reducing resource utilization and communication load in multi-agent systems, reducing dependence on online learning, and improving system stability and efficiency.
Smart Images
Figure CN120469216A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of multi-agent control technology, and in particular to a neural network control method, a controller construction method and a system for multi-agent collaboration. Background Art
[0002] Discrete-time strict feedback multi-agent systems are widely used in scenarios such as drone formations and robot collaboration. The continuous-time control scheme of discrete-time strict feedback multi-agent systems may cause future states to appear in the control law, thus causing non-causality problems.
[0003] Solutions to non-causal problems in multi-agent control systems include n-step-ahead prediction technology, dynamic neural network prediction methods, and variable replacement technology based on backstepping.
[0004] However, in real-world multi-agent systems, system state variables are often difficult to obtain. As the system order increases and the impact of nonlinear control inputs increases, the number of nodes and complexity of the neural network required for training increases dramatically. This leads to the need for higher communication bandwidth between agents in multi-agent control systems.
[0005] In order to solve the distributed collaborative learning problem of discrete-time strict feedback multi-agent systems under directed graphs, the present application provides a neural network predictive control method for multi-agent collaborative systems. Summary of the Invention
[0006] To overcome the problems existing in the related art, the first aspect of the present application provides a multi-agent collaborative neural network control method, comprising the following steps:
[0007] Obtain a first state vector; the first state vector is the first state vector of the kth time step of the i-th agent;
[0008] Predicting based on the first state vector, obtaining a second state vector; the second state vector is the state vector of the i-th agent at the k+n-th time step;
[0009] Predicting based on the second state vector to determine an expected trajectory; the expected trajectory is the reference trajectory of the i-th agent at the k+n-th time step;
[0010] The desired trajectory is input into a pre-built neural network controller, which outputs the system control input of the k+nth time step of the i-th agent; the weight vector of the neural network controller is trained and updated based on the trajectory estimation error of the i-th agent.
[0011] In one embodiment, the neural network controller is
[0012]
[0013] Among them, u i (k) is the system control input, y i,d (k+n) is the k+nth reference trajectory of the i-th agent; g i is the known system gain; is the neural network weight vector obtained by training the trajectory errors of i agents; It is the nonlinear system function fitted by the neural network.
[0014] In one embodiment, the neural network weight vector is defined as:
[0015]
[0016] Among them, [k a ,k b ] represents the time period when the neural network weights enter the steady state process, is the neural network weight update law.
[0017] A second aspect of the present application provides a method for constructing a multi-agent collaborative neural network controller, comprising the following steps:
[0018] S100, determining a state vector predictor of the i-th agent through a strict feedback multi-agent system;
[0019] S200, constructing an n-step-ahead predictor based on the state vector predictor, and predicting a reference trajectory;
[0020] S300, determining an estimation error of the i-th agent according to a difference between the reference trajectory and the expected trajectory;
[0021] S400, designing a neural network weight update law for the i-th intelligent agent based on the estimation error of the i-th intelligent agent, and constructing a neural network controller to be trained;
[0022] S500, iterating the neural network controller to be trained according to the neural network weight update law of the i-th intelligent agent;
[0023] S600: Use the iterated neural network controller as the offline controller of the i-th intelligent agent.
[0024] In one embodiment, after S500, the method further includes:
[0025] S501. Communication between agents is performed based on the update rate of the neural network weights of the i-th agent and the update rate of the neural network weights of the j-th agent.
[0026] In one embodiment, S400 specifically includes:
[0027] S401, according to the update rate of the neural network weight of the i-th agent, sum and average to obtain a stable update rate of the neural network weight;
[0028] S402. Design an input controller based on the update rate of the neural network weights of the i-th agent.
[0029] In one embodiment, after S402, the method further includes:
[0030] S403. Design an offline controller according to the update rate of the neural network weights.
[0031] In one embodiment, S300 specifically includes:
[0032] A state vector predictor is used to track the system state;
[0033] Determine the ideal error estimator from the system state;
[0034] The estimated error can be obtained by subtracting the constructed error estimator from the ideal error estimator.
[0035] In one embodiment, S402 specifically includes:
[0036] A neural network weight update law of the input controller is constructed based on the estimated error.
[0037] The third aspect of the present application provides a multi-agent collaboration system that performs multi-agent collaboration control based on the steps in the method described in the first aspect of the present application.
[0038] The technical solution provided by this application may have the following beneficial effects:
[0039] During the training process of the neural network controller, a stable neural network weight update rate is obtained by summing and averaging the update rates of the neural network weights of i agents. After iterative stabilization, the neural network controller is designed based on the neural network weight update rate. The neural network controller provided in this application can output the current system control input based on the reference trajectory of the k+nth step. This neural network controller can effectively reduce resource utilization, eliminating the need for repetitive design of control tasks through online learning, and reducing the AC load of the multi-agent system.
[0040] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.
[0042] Figure 1 A flowchart of a neural network control method according to an embodiment of the present application is shown;
[0043] Figure 2 A flowchart of a method for constructing a neural network controller according to an embodiment of the present application is shown;
[0044] Figure 3 This is a schematic diagram of the data structure of a neural network controller shown in an embodiment of the present application;
[0045] Figure 4 This is the expected tracking effect diagram of the controller based on the adaptive weight update law;
[0046] Figure 5 This is a diagram showing the expected tracking effect of the controller according to an embodiment of the present application;
[0047] Figure 6 This is a diagram showing the weight convergence effect of the controller in an embodiment of the present application. DETAILED DESCRIPTION
[0048] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.
[0049] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
[0050] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0051] Example 1
[0052] In order to solve the distributed collaborative learning problem of a discrete-time strict feedback multi-agent system under a directed graph, the present invention provides a neural network control method for a multi-agent collaborative system, such as Figure 1 As shown, the following steps are included:
[0053] S1. Obtain a first state vector;
[0054] S2. Predicting based on the first state vector to obtain a second state vector;
[0055] S3. Determine a desired trajectory based on the second state vector prediction;
[0056] S4. Input the desired trajectory into a pre-built neural network controller, and output the system control input of the k+nth time step of the i-th agent;
[0057] In an embodiment of the present application, the first state vector is the first state vector of the i-th intelligent agent at the k-th time step; the second state vector is the state vector of the i-th intelligent agent at the k+n-th time step; and the expected trajectory is the reference trajectory of the i-th intelligent agent at the k+n-th time step.
[0058] In an embodiment of the present application, the weight vector of the neural network controller is updated by the trajectory errors of i agents.
[0059] Specifically, the neural network controller is
[0060]
[0061] Among them, u i (k) is the system control input of the i-th agent, y i,d (k+n) is the reference trajectory of the k+nth step of the i-th agent; g i is the known system gain of the i-th agent; is the neural network weight vector obtained by training the trajectory error of the i-th agent; It is the nonlinear system function fitted by the neural network.
[0062] In the embodiment of the present application, the neural network weight vector is defined as:
[0063]
[0064] Among them, [k a ,k b ] represents the time period when the neural network weights enter the steady state process, is the neural network weight update law.
[0065] Specifically, the neural network weight update law is designed based on the trajectory errors of i agents. By adopting the designed neural network weight update law, the neural network weights of multiple agents can become an exponentially convergent sequence.
[0066] In the embodiment of the present application, the neural network weight update law is:
[0067]
[0068] in, is the estimated error between the expected trajectory and the predicted trajectory of the ith agent at the k+1th time step, r and τ are designed parameters and are positive numbers; let A = [a ij ] N×N Represents the neighbor matrix, where if a ij =1, it means that the i-th agent can obtain information from the j-th agent.
[0069] During the training process of the neural network controller, a stable neural network weight update rate is obtained by summing and averaging the update rates of the neural network weights of the i-th agent. After the iterations are stabilized, the neural network controller is designed based on the update rate of the neural network weights.
[0070] The neural network controller described in the embodiments of the present application can output the current system control input based on the reference trajectory of the k+nth step. This neural network controller can effectively reduce resource utilization, eliminate the need for repetitive design of control tasks through online learning, and reduce the communication load of the multi-agent system.
[0071] Example 2
[0072] Based on the neural network control method of Example 1, the embodiment of this application specifically describes the construction principle of the neural network controller.
[0073] The present invention provides a method for constructing a multi-agent collaborative neural network controller. Figure 2 As shown, the following steps are included:
[0074] S100, determining a state vector predictor of the i-th agent through a strict feedback multi-agent system;
[0075] S200, constructing an n-step-ahead predictor based on the state vector predictor, and predicting a reference trajectory;
[0076] S300, determining an estimation error of the i-th agent according to a difference between the reference trajectory and the expected trajectory;
[0077] S400, designing a neural network weight update law for the i-th intelligent agent based on the estimation error of the i-th intelligent agent, and constructing a neural network controller to be trained;
[0078] S500, iterating the neural network controller to be trained according to the neural network weight update law of the i-th intelligent agent;
[0079] S600: Use the iterated neural network controller as the offline controller of the i-th intelligent agent.
[0080] After S500, it also includes:
[0081] S501, based on the update rate of the neural network weight of the i-th agent and the update rate of the neural network weights of the jth agent Conduct communication between intelligent agents.
[0082] The neural network controller of the embodiment of the present application is used to solve the control system problem where the system state is unpredictable. Under the condition of switching the directional communication topology structure, the design method of the leaderless following multi-agent learning control system is used to further illustrate the construction process of the neural network controller.
[0083] In S100, a controller is constructed for the following control system model:
[0084]
[0085] in, is the state vector of the i-th agent, u i (k)∈R is the control input of the i-th agent, y d (k)∈R is the regression reference trajectory; the gain g j satisfy:
[0086] It is understandable that the function Is an unknown nonlinear system function that satisfies the parameterization conditions of the agent:
[0087]
[0088] Among them, the matrix W i ∈R q×m is the unknown parameter matrix, function S i :R n ×[0,∞)→R q is a known regression function.
[0089] Specifically, the constructed state vector predictor is as follows:
[0090]
[0091] Substitute the state vector of the j+1th generation agent into the equation of the jth generation agent state vector and iterate. The state vector of the k+nth time step of the ith agent can be expressed as:
[0092]
[0093] Among them, x i,1 (k+n) is the k+Nth step state vector of the first generation of the i-th agent; is the unknown nonlinear system function of the first generation of the i-th agent at the k+n-1th step; g1x 1,2 (k+n-1) is the state vector of the second generation (k+N-1th step) of the i-th agent; g1 is the known gain.
[0094] In S200, an n-step-ahead predictor is constructed.
[0095] Furthermore, the state input in S100 is transformed to obtain the following input-output predictor:
[0096]
[0097] Among them, the introduction of new variables , define the control input as , using the n-step-ahead prediction technique, we get the following output feedback n-step-ahead predictor:
[0098] In the embodiment of the present application, by defining a new variable is the input excitation vector of the neural network, and Contains all input and output prediction items and control input items u i (k), Substitute into the unknown nonlinear system function that can be fitted by the neural network You can get:
[0099] In steps S300 to S500, the neural network weight update law is designed by estimating the error of the n-step-ahead predictor.
[0100] S300 specifically includes the following steps:
[0101] In a neural network-based predictive controller, the ideal controller has the form:
[0102]
[0103] Among them, W i *T is the transpose of the ideal weight vector.
[0104] In an ideal controller, is unknown, so the ideal weight vector determines the accuracy of the controller. In order to obtain an ideal weight vector that is more consistent with the above control system, the actual controller designed in the embodiment of the present application is:
[0105]
[0106] Among them, u i (k) is the system control input; y i,d (k+n) is the k+nth reference trajectory of the i-th agent; g i is the known system gain; the estimated weight vector of the neural network Used to estimate the ideal weight vector and
[0107] In existing neural network controllers, the error estimator is usually designed as:
[0108] e i (k) = x i,j (k)-y i,d (k)
[0109] Among them, e i (k) is the error estimate.
[0110] In the embodiment of the present application, a state vector predictor is used to track the system state. Therefore, the ideal error estimator is in the following mode:
[0111]
[0112] Introducing the actual neural network weight vector Let k1 = k-n+1, then the error estimator is:
[0113]
[0114] In the embodiment of the present application, the error estimator Used to estimate the ideal error estimator e i,y (k+n). By taking the difference between the error estimator and the ideal error estimator, we can get:
[0115]
[0116] in,
[0117] In the embodiment of the present application, based on the error estimator Constructing a neural network update law. S400 specifically includes:
[0118] S401, according to the update rate of the neural network weight of the i-th agent, sum and average to obtain a stable update rate of the neural network weight;
[0119] S402, designing an input controller according to the update rate of the neural network weights of the i-th agent;
[0120] S403. Design an offline controller according to the update rate of the neural network weights.
[0121] The update law of the neural network based on the discrete strict feedback system is designed as follows:
[0122]
[0123] Among them, r and τ are the design parameters and are positive numbers. Let A=[a ij ] N×N Represents the neighbor matrix, where if a ij =1, it means that the i-th agent can obtain information from the j-th agent.
[0124] In the existing update law, the ideal error estimator will cause the neural network weights to fail to become an exponentially convergent sequence.
[0125] Therefore, in step S402, a neural network weight update law of the input controller is constructed based on the estimation error.
[0126] The following neural network weight update rate is obtained:
[0127]
[0128] In the embodiment of the present application, the neural network weight update law of the present design can make the neural network weights of multiple agents become an exponentially convergent sequence, and the data structure of the weight update is as follows: Figure 3 shown.
[0129] Figure 4 and Figure 5Comparison diagrams of the output trajectories and expected trajectories of the multi-agent trained based on the adaptive weight update law and the multi-agent of the embodiment of the present application, respectively.
[0130] exist Figure 4 Among the four intelligent controllers trained with the adaptive weight update law, their output trajectories gradually match the expected trajectories within 5000 time steps.
[0131] exist Figure 5 Among the four intelligent agent controllers provided in the embodiment of the present application, within 3000 time steps, their output trajectories are better aligned with the expected trajectories, and there is no obvious output fluctuation.
[0132] Based on the neural network weight update law, a stable neural network weight is obtained through training, and the neural network controller is designed as follows:
[0133]
[0134] in, is the stable neural network weight obtained after training, which is defined as follows:
[0135]
[0136] Among them, [k a ,k b ] represents the time period when the neural network weights enter the steady state process.
[0137] Furthermore, the error estimator of the embodiment of the present application is calibrated by the Lyapunov function. and the neural network weight update law Perform stability analysis and proof.
[0138] Specifically, the Lyapunov function is selected as follows:
[0139]
[0140] Among them, Γ -1 Expressed as a designed positive definite matrix, and satisfying λ max (Γ)<ε, ε is a positive parameter of the design.
[0141] By differentiating V(k), we can obtain:
[0142]
[0143] Substituting the error estimator and the neural network weight update law into ΔV, we can obtain:
[0144]
[0145] in, L represents the Laplacian matrix of the communication graph between agents, ||S(k)|| 2 ≤l,S(k)ΓS T (k)≤rl,r,l are design parameters.
[0146] Using Young's inequality, we get the following conclusions:
[0147]
[0148] Introducing these conclusions into the above results, ΔV can be expressed as follows:
[0149]
[0150] By choosing appropriate design parameters: And there is The following results can be obtained:
[0151]
[0152] If exists In all processes, 0≤V(k)≤V(k-1)-Ψ, show is bounded, ΔV is negative, that is, and It's very tiny. exists in So all the system states are bounded.
[0153] like Figure 6 It shows that the weight value of the controller provided in the embodiment of the present application can converge to a stable value, indicating that the neural network control of the embodiment of the present application can reach a balanced state through the above construction method.
[0154] Example 3
[0155] A multi-agent collaborative system performs multi-agent collaborative control based on the steps in the neural network control method of embodiment one.
[0156] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.
[0157] The solution of the present application has been described in detail above with reference to the accompanying drawings. In the above embodiments, the description of each embodiment has its own focus. For parts not described in detail in a particular embodiment, please refer to the relevant description of other embodiments. Those skilled in the art should also be aware that the actions and modules mentioned in the description are not necessarily required for this application.
[0158] In addition, it can be understood that the steps in the method of the embodiment of the present application can be adjusted in order, merged and deleted according to actual needs, and the modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.
[0159] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.
[0160] Alternatively, the present application can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or electronic device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.
[0161] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the application herein may be implemented as electronic hardware, computer software, or combinations of both.
[0162] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems and methods according to multiple embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0163] The embodiments of the present application have been described above. The above description is illustrative and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to the technology in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.
Claims
1. A multi-agent collaborative neural network control method, characterized in that: The following steps are involved: Obtain a first state vector; the first state vector is the first state vector of the kth time step of the i-th agent; Predicting based on the first state vector, obtaining a second state vector; the second state vector is the state vector of the i-th agent at the k+n-th time step; Predicting based on the second state vector to determine an expected trajectory; the expected trajectory is the reference trajectory of the i-th agent at the k+n-th time step; The desired trajectory is input into a pre-built neural network controller, which outputs the system control input of the k+nth time step of the i-th agent; the weight vector of the neural network controller is trained and updated based on the trajectory estimation error of the i-th agent.
2. A multi-agent collaborative neural network control method according to claim 1, characterized in that: The neural network controller is Among them, u i (k) is the system control input, y i,d (k+n) is the k+nth reference trajectory of the i-th agent; g i is the known system gain; is the neural network weight vector obtained by training the trajectory errors of i agents; It is the nonlinear system function fitted by the neural network.
3. A multi-agent collaborative neural network control method according to claim 2, characterized in that: The neural network weight vector is defined as: Among them, [k a ,k b ] represents the time period when the neural network weights enter the steady state process, is the neural network weight update law.
4. A method for constructing a multi-agent collaborative neural network controller, characterized in that: The following steps are involved: S100, determining a state vector predictor of the i-th agent through a strict feedback multi-agent system; S200, constructing an n-step-ahead predictor based on the state vector predictor, and predicting a reference trajectory; S300, determining an estimation error of the i-th agent according to a difference between the reference trajectory and the expected trajectory; S400, designing a neural network weight update law for the i-th intelligent agent based on the estimation error of the i-th intelligent agent, and constructing a neural network controller to be trained; S500, iterating the neural network controller to be trained according to the neural network weight update law of the i-th intelligent agent; S600: Use the iterated neural network controller as the offline controller of the i-th intelligent agent.
5. A method for constructing a multi-agent collaborative controller according to claim 4, characterized in that: After S500, it also includes: S501. Communication between agents is performed based on the update rate of the neural network weights of the i-th agent and the update rate of the neural network weights of the j-th agent.
6. A method for constructing a multi-agent collaborative controller according to claim 4, characterized in that: S400 specifically includes: S401, according to the update rate of the neural network weight of the i-th agent, sum and average to obtain a stable update rate of the neural network weight; S402. Design an input controller based on the update rate of the neural network weights of the i-th agent.
7. A method for constructing a multi-agent collaborative controller according to claim 6, characterized in that: After S402, it also includes: S403. Design an offline controller according to the update rate of the neural network weights.
8. A method for constructing a multi-agent collaborative controller according to claim 7, characterized in that: S300 specifically includes: A state vector predictor is used to track the system state; Determine the ideal error estimator from the system state; The estimated error can be obtained by subtracting the constructed error estimator from the ideal error estimator.
9. A method for constructing a multi-agent collaborative controller according to claim 8, characterized in that: S402 specifically includes: A neural network weight update law of the input controller is constructed based on the estimated error.
10. A multi-agent collaborative system, characterized in that: Multi-intelligent collaborative control is performed based on the steps in the method described in any one of claims 1 to 3.