Agent security control network training method and control method, electronic device, and medium

Abstract

The application provides an agent safety control network training method, a control method, an electronic device and a medium, and relates to the technical field of agents; the training method comprises: obtaining a training sample set; inputting the training sample set into a pre-constructed heterogeneous graph control barrier function network, outputting barrier function values corresponding to each agent, and updating parameters of the heterogeneous graph control barrier function network based on a preset first loss function; inputting the training sample set into a task controller to output an action sequence; and correcting the action sequence through quadratic programming under a forward safety constraint model to obtain a centralized safety action sequence of each agent; taking the centralized safety action sequence as a supervision label and training a distributed control network together with the training sample set; and repeatedly executing the above steps until a preset iteration termination condition is met. The application enables each agent to output a high-reliability safety control action only by relying on local observation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent agents, in particular to an intelligent agent safety control network training method, a control method, an electronic device and a medium. BACKGROUND

[0002] With the rapid development of artificial intelligence and autonomous system technology, multi-agent collaborative systems composed of multiple intelligent agents show broad application prospects in intelligent manufacturing, warehouse logistics, unmanned clusters, smart cities and other scenarios. In these systems, intelligent agents often have heterogeneity, that is, different types of intelligent agents (such as ground mobile robots, unmanned aerial vehicles, robotic arms, etc.) have significant differences in dynamic models, perception capabilities, motion constraints and task objectives. This heterogeneity, while enhancing the overall functional diversity of the system, also poses serious challenges to collaborative control and safety assurance.

[0003] In the prior art, the execution actions of intelligent agents are constrained by control barrier functions (CBF) so that they can perform tasks within their respective safety ranges.

[0004] However, the classical CBF method requires manual design of barrier functions, and has certain limitations in the face of high-dimensional, non-convex, heterogeneous and other complex multi-agent scenarios. The reliability of manually designed control barrier functions is low, which in turn leads to low reliability of the generated intelligent agent execution actions. SUMMARY

[0005] Therefore, the purpose of the present application is to provide an intelligent agent safety control network training method, a control method, an electronic device and a medium to improve the reliability of the generated intelligent agent execution actions.

[0006] In a first aspect, an intelligent agent safety control network training method is provided, applied to a multi-agent collaborative system composed of at least two types of heterogeneous intelligent agents; the method comprises:

[0007] Obtaining a training sample set, each set of training samples in the training sample set containing trajectory expansion state data of each intelligent agent and its neighbors at a certain sampling time;

[0008] Inputting the training sample set into a pre-constructed heterogeneous graph control barrier function network, outputting the barrier function values corresponding to each intelligent agent, and updating the parameters of the heterogeneous graph control barrier function network based on a pre-set first loss function;

[0009] Constructing a forward safety constraint model according to the barrier function values corresponding to each intelligent agent and the gradient of the barrier function values, the forward safety constraint model being used to control each intelligent agent to always be within a safety range;

[0010] inputting the training sample set into the task controller to output an initial action sequence; and correcting the action sequence under a forward safety constraint model through quadratic programming to obtain a centralized safety action sequence of each agent; the centralized safety action sequence is used to control all agents to complete a target task and be within a safety range;

[0011] inputting the centralized safety action sequence as a supervision label into a pre-built distributed control network together with the training sample set to output a distributed action sequence of each agent, and updating network parameters of the distributed control network based on a pre-built second loss function; the distributed action sequence is used to control the corresponding agent to be within the safety range;

[0012] repeating the above steps until a preset iteration termination condition is met.

[0013] Optionally, obtaining the training sample set comprises:

[0014] under a preset operation strategy, running the multi-agent collaboration system, and collecting initial trajectory state data of each agent and neighbors thereof according to a preset sampling period; the neighbors are other agents observed by each agent within a preset observation range;

[0015] dimensionally extending the initial trajectory state data of each agent and neighbors thereof to obtain trajectory extended state data, so as to unify the state dimensions of heterogeneous agents;

[0016] storing the trajectory extended state data of each agent and neighbors thereof in the sample pool in a graph structure; node features of the graph structure include types of the agents and safety radii; edge features of the graph structure include relative states and relative safety radii between each agent and the neighbors;

[0017] periodically randomly extracting a preset number of training samples from the sample pool to form the training sample set.

[0018] Optionally, the method further comprises:

[0019] updating the initial trajectory state data of each agent and neighbors thereof based on the distributed action sequence output by the distributed control network;

[0020] dimensionally extending the updated trajectory state data and storing the trajectory state data in the sample pool in a graph structure.

[0021] Optionally, the first loss function is:

[0022]

[0023] wherein, is a preset safety margin; is the trajectory extended state data of each agent and neighbors thereof; a safety range set of each agent; an output barrier function value; a gradient of the barrier function value; an extended class a function.

[0024] Optionally, the forward safety constraint model is:

[0025]

[0026] wherein, an output barrier function value; a gradient of the barrier function value; an extended class a function.

[0027] Optionally, the trajectory extended state data of each agent and its neighbors are input to the distributed control network in the form of a heterogeneous local graph, the heterogeneous local graph including a center node, neighbor nodes and edges, and corresponding information including center node features, neighbor node features and edge features; the process of the distributed control network outputting a distributed action sequence of each agent includes:

[0028] For each heterogeneous local graph, the center node features and the neighbor node features thereof are respectively linearly mapped to obtain initial embedding representations of each node;

[0029] Based on the initial embedding representations of each node, a node attention mechanism is used to calculate attention weights of each neighbor node feature relative to the center node feature; and based on the attention weights of each neighbor node feature, the neighbor node features are weightedly fused to obtain advanced features of the neighbor nodes;

[0030] An edge attention mechanism is used to calculate edge attention weights of each edge feature, and based on the edge attention weights, each edge feature is weightedly fused to obtain edge advanced features;

[0031] The advanced features of the neighbor nodes, the edge advanced features and the center node features are spliced to form joint features;

[0032] The joint features are input to a multi-layer perception machine to output the distributed action sequence of each agent.

[0033] Optionally, updating the network parameters of the distributed control network based on the pre-constructed second loss function includes:

[0034] According to the deviation between the distributed action sequence of each agent and the centralized safety action sequence, an individual loss value of each agent is determined;

[0035] The individual loss values of each agent are weightedly fused to obtain a total loss value of the second loss function;

[0036] Based on the total loss value, the network parameters of the distributed control network are updated by gradient back propagation.

[0037] In a second aspect, a method for safe control of agents is provided, which is applied to a multi-agent cooperation system composed of at least two types of heterogeneous agents; the method comprises:

[0038] Real-time trajectory expansion state data of the target agent and its neighbors are taken; the neighbors are other agents observed by each agent within a preset observation range;

[0039] The trajectory expansion state data is input into the distributed control network trained by the method of any one of the first aspect to output a distributed action sequence of the target agent.

[0040] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

[0041] The memory is used to store a computer program;

[0042] The processor is used to execute the program stored on the memory, and implement the method of any one of the first aspect or the second aspect.

[0043] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of the first aspect or the second aspect.

[0044] The present application provides a method for training a safe control network of agents, a control method, an electronic device and a medium, proposes a learnable control barrier function network based on a graph neural network, which can automatically learn a barrier function suitable for heterogeneous multi-agents from multi-agent interaction data; on this basis, the safe constraints output by the control barrier function network are embedded in a quadratic programming to generate a centralized safe action sequence, which is used as a supervision signal to train a distributed control network, so that each agent can still output a safe control action with high reliability under the condition of only relying on local observation; and the bottleneck of manually designing a barrier function is effectively overcome.

[0045] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0047] Figure 1 A flow chart of a method for training an agent safety control network is shown;

[0048] Figure 2 A diagram structure diagram of a trajectory expansion state of an agent and its neighbors is shown;

[0049] Figure 3 A structure diagram of an hGCBF network is shown;

[0050] Figure 4 A structure diagram of a distributed control network is shown;

[0051] Figure 5 A training framework diagram of joint training of an hGCBF network and a distributed control network is shown;

[0052] Figure 6 A flow diagram of an agent safety control method is shown;

[0053] Figure 7 A structure diagram of an electronic device is shown. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application.

[0055] Considering that the classical CBF method needs to manually design the barrier function, has certain limitations in the face of high-dimensional, non-convex, heterogeneous and other complex multi-agent scenes, the reliability of manually designing the control barrier function is low, and then the reliability of the generated agent execution action is low.

[0056] Based on this, the embodiment of the application provides an agent safety control network training method, which is described below through an embodiment.

[0057] The embodiment of the application provides an agent safety control network training method, which is applied to a multi-agent cooperation system composed of at least two types of heterogeneous agents. For example, in the scene of intelligent logistics warehousing, an automated guided vehicle (AGV, Automated Guided Vehicle) and an omnidirectional mobile robot cooperate.

[0058] As shown in the method, the method comprises the following steps: Figure 1

[0059] Step S101: Obtain a training sample set, and each set of training samples in the training sample set contains trajectory expansion state data of each agent and its neighbors at a certain sampling time.

[0060] Among them, the neighbors are other agents observed or perceived by each agent within a preset observation range.

[0061] In a feasible implementation manner, obtaining the training sample set comprises:

[0062] Step S101A: Under a preset running strategy, run the multi-agent cooperation system, and collect initial trajectory state data of each agent and its neighbors according to a preset sampling period.

[0063] In this step, the preset running strategy is, for example, advancing towards a target point at a constant speed, etc.

[0064] Under the running strategy, a pre-set task controller is called to generate an action sequence of each agent, and the agent performs actions according to the action sequence until the target point is reached, and the initial trajectory state data of each agent in the running process is collected.

[0065] Among them, the task controller is a controller that only focuses on the task target and does not guarantee safety.

[0066] For example, the task controller can be a PID controller, etc., and the control function of the task controller is as follows:

[0067] (1);

[0068] Among them, is a preset running strategy; is an agent​ trajectory state data of the multi-agent system at time t0; for the intelligent agent .

[0069] In one specific example, the process of collecting initial trajectory state data is as follows:

[0070] First, at time t0, when the multi-agent cooperative system has not yet started running, the initial trajectory state data of the intelligent agent and its neighbors is collected .

[0071] Second, run the simulation environment , input the initial trajectory state data into the control function to obtain the action sequence of the intelligent agent at time t1 ;

[0072] Third, based on the action sequence at the current time and the dynamic model of the intelligent agent , the initial trajectory state data of the next time is collected .

[0073] Among them, for the intelligent agent in the heterogeneous multi-agent system, its dynamic motion model can be expressed as:

[0074] (2)

[0075] In the formula, is the inherent state parameter of the intelligent agent , such as inertia, size, etc.; is the dynamic parameter in the running process, such as speed, motion direction, etc.; is the action sequence of the intelligent agent at time t1 , wherein is the time t1 ; is the trajectory state data of the intelligent agent and its neighbors at time t1 ; is the trajectory state data of the intelligent agent and its neighbors at time t1 .

[0076] Repeat the second and third steps until the intelligent agent reaches the target point. Get the initial trajectory state data of all sampling times of each intelligent agent and its neighbors.

[0077] ​​Step S101B: Dimension expansion is performed on the initial trajectory state data of each agent and its neighbors to obtain trajectory extended state data, so as to unify the state dimensions of heterogeneous agents.

[0078] In this step, for heterogeneous agents with different state dimensions and action dimensions, a unified dimension can be constructed by a zero padding method. Specifically, the state dimension of each agent is expanded to the maximum value of the state dimension in all agents to obtain an extended state space The maximum value of the state dimension in all agents.

[0079] For example, in a heterogeneous multi-agent system: the state dimension of the agent AVG is 3, such as wherein, is the position of the agent AVG; is the angle of AVG. The state dimension of the omnidirectional mobile robot is 4, such as wherein, is the position of the omnidirectional mobile robot; is the velocity of the omnidirectional mobile robot;

[0080] Due to the different dimensions, they cannot be directly processed by the same control barrier function or safety control network.

[0081] In the embodiment of the application, the dimensions are unified to a maximum dimension of 4 by zero padding, and the missing dimensions of the low-dimensional agent are supplemented with 0. Specifically, the original trajectory state of the agent AGV is: and after expansion: .

[0082] Similarly, the action sequence is expanded to The maximum value of the action sequence dimension in all agents, is the action sequence of all agents.

[0083] Correspondingly, the dynamic motion model of the expanded agent is: (3);

[0084] wherein, is the expanded intrinsic state parameter of the agent , such as inertia, size, etc.; is the dynamic parameter in the expanded running process, such as velocity, motion direction, etc.; is the expanded action sequence of the agent at time , wherein, at time​​ ; for At any moment, intelligent agent The trajectory extension status data of the vehicle and its neighbors; for At any moment, intelligent agent The trajectory extension state data of the vehicle and its neighbors, In , ( The maximum value of the state of degree among all agents; (Extend state data for the trajectories of all agents and their neighbors) 。

[0085] This extension method ensures that the extended intelligent agent The front of the dynamic motion model Dimensional components ( For intelligent agents The dimension values ​​of the trajectory state data are kept consistent with the data before expansion, thereby ensuring the equivalence of the expanded dynamic motion model with the original dynamic motion model.

[0086] In other words, any state trajectory of the original system is related to the previous state trajectory of the extended system. The components are strictly equivalent.

[0087] Step S101C: Store the trajectory extended state data of each agent and its neighbors in the sample pool in the form of a graph structure.

[0088] like Figure 2 As shown, the graph structure includes node features and edge features; the node features include the node type encoding safety radius (i.e., the distance at which an agent does not conflict or collide with other agents); this type encoding is composed of the node type and the type of agent (e.g., drone, unmanned vehicle).

[0089] In terms of edge feature design, edge features consist of two elements: relative safety radius and relative agent state.

[0090] In heterogeneous multi-agent systems, different types of agents (such as ground robots, drones, and robotic arms) often have inconsistent state space dimensions and physical meanings. For example, drones contain altitude information while ground robots do not, and some agents have velocity states while others only provide position. This heterogeneity in state representation directly leads to difficulties in unified interaction modeling between agents, especially when facing compatibility challenges in constructing the edge features required for graph neural networks.

[0091] To address this problem, embodiments of the present invention propose a unified method for representing relative states:

[0092] Firstly, the states of all agents are systematically disassembled according to the predefined global coordinate axes (such as x, y, z positions and corresponding velocity components);

[0093] Secondly, when calculating the relative state between any two agents, if one party is missing a state component due to type or capability limitations (such as ground robots without z coordinates), zero values are automatically filled in at the corresponding coordinate axis positions;

[0094] Finally, based on this aligned state vector, the relative position and relative velocity between the two are calculated and used as relative state elements, along with the relative safety radius (a safety distance parameter determined by the types of both parties) to form the edge features.

[0095] Through the above design, the edge features always have fixed dimensions and consistent semantics, regardless of how the agents are combined. This not only effectively solves the problem of state space mismatch in heterogeneous systems, but also ensures that the graph neural network can uniformly and learnably model the interaction between different types of agents during message passing, thereby significantly improving the compatibility and generalization ability of the system.

[0096] Through the above graph construction process, the trajectory expansion state data of each agent and its neighbors form a heterogeneous local subgraph.

[0097] Step S101D: periodically randomly extract a predetermined number of training samples from the sample pool to form a training sample set.

[0098] For example, in the simulation, 10 rounds were run, each round had 100 time steps, and there were 5 agents per step. The total number of original samples ≈ 10 x 100 x 5 = 5000 local observations.

[0099] All these samples are stored in the sample pool. When training, assuming that batch_size = 64, then 64 local observation samples are randomly extracted from the buffer for the first time. In later training, for example, every 10 iterations, new samples are re-extracted from the sample pool and added to the current samples.

[0100] This periodic mixing of new and old samples helps to more comprehensively cover the state and action space under different strategies, avoiding overfitting to the current strategy and improving the robustness of the system.

[0101] Step S102: input the training sample set into the pre-constructed heterogeneous graph control barrier function network, output the barrier function values corresponding to each agent, and update the parameters of the heterogeneous graph control barrier function network based on the pre-set first loss function.

[0102] In classical control theory, the control barrier function CBF is usually artificially constructed analytically, so the constructed control barrier function is an explicit, fixed, known mathematical expression. Only applicable to simple scenarios (such as isomorphic scenarios), and complex heterogeneous systems cannot be designed manually.

[0103] Therefore, the embodiment of the present application learns an implicit control barrier function through the learning mode of a neural network. Specifically, first, a hGCBF (Heterogeneous Graph Control Barrier Function) network is constructed, the input of the network is a local subgraph centered on a heterogeneous agent i , and the output is a real number . The network structure is a graph neural network, and the learning goal is to make the hGCBF network satisfy two constraint conditions of the classical CBF:

[0104] One is static classification:

[0105] Safe state: (4) and dangerous state: (5);

[0106] The other is a dynamic differential constraint: (6);

[0107] Wherein, is the output barrier function value; is the gradient of the barrier function value; is an extended class function. The dynamic differential constraint guarantees the forward invariance of the safe state.

[0108] Through iterative learning, the hGCBF network can provide a unified implicit control barrier function for various types of agents, meeting the needs of heterogeneous scenarios.

[0109] Further, the network structure of the hGCBF network is as shown in Figure 3 , and the hGCBF network includes an input layer, an initial multilayer perceptron, an attention module, a summation module, and a final multilayer perceptron. The working process of the network result is as follows:

[0110] Step S102A: construct a local subgraph centered on the target agent as input;

[0111] The input includes three basic elements: the node features of the target agent itself, the node features of its neighbor agents, and the edge features connecting the target agent and each neighbor.

[0112] Step S102B: The three types of inputs are respectively encoded by independent initial multi-layer perceptrons, generating the self-embedding and neighbor embedding of the agent.

[0113] Step S102C: Calculate the importance weight of the neighbor nodes to the center node.

[0114] With the self-embedding and each neighbor embedding of the target agent as input, the attention weight of each neighbor to the current safety state is dynamically evaluated through a learnable attention mechanism; then, the neighbor embeddings are weighted and summed according to the obtained attention weights to obtain the aggregated neighbor information.

[0115] Step S102D: Weighted summation

[0116] The self-embedding of the target agent and the aggregated neighbor information are fused (e.g., spliced or added) to obtain high-level features.

[0117] Step S102E: The high-level features are sent into the final multi-layer perceptron to be mapped into a single real value output, i.e., the barrier function value.

[0118] The entire hGCBF network is completely differentiable, supports end-to-end training through backpropagation, and can automatically learn the static classification boundary and dynamic differential constraint required to meet the control barrier function theory in the heterogeneous multi-agent interaction scenario.

[0119] The barrier function value output by the above hGCBF network and the two constraint conditions of the classical CBF are used to construct a first loss function, which is as follows:

[0120] (7);

[0121] wherein, is a preset safety margin; is the trajectory expansion state data of each agent and its neighbors; is the output barrier function value; is the gradient of the barrier function value; is the expansion class function; is the safety range set of each agent.

[0122] (8);

[0123] wherein, denotes the joint state vector of multiple agents; denotes the joint state vector; denotes the position of agent ; denotes the position of agent the position of the agent; represents the safety radius of the agent represents the safety radius of the agent represents the safety radius of the agent represents the safety radius of the agent precisely defines the state region in which all agents do not collide with each other.

[0124] According to the above embodiment, it can be understood that the extended dynamic motion model is equivalent to the original dynamic motion model, which means that the safety set of the extended system is equivalent to the safety set of the original system , that is, as long as the forward invariance of is guaranteed, the safety of the original system is guaranteed.

[0125] Specifically,

[0126] This constraint restricts each agent within the safety range, and if it is not satisfied, a penalty term will be generated.

[0127] By setting the safety margin , a buffer zone on the classification boundary is introduced, which forces the hGCBF network to establish a clear and robust decision boundary between safe and dangerous states, avoiding the model output close to zero (h=0), thereby improving the training stability and deployment safety. On the other hand, by setting , a certain range of label uncertainty can be tolerated, avoiding the model from being difficult to converge due to boundary sample oscillation.

[0128]

[0129] This constraint restricts each agent within the safety range and if it is not satisfied, a penalty term will be generated.

[0130] Similarly,

[0131] The safety range set of the agent is constrained to maintain forward invariance, which means that at any future time, the agent clock is within the safety range, and if it is not satisfied, a penalty term will be generated.

[0132] Through the first loss function, the parameters of the hGCBF network are adjusted through backpropagation, and the goal is to minimize the loss of the first loss function, that is, to converge to 0.

[0133] Step S103: Construct a forward safety constraint model based on the obstacle function value and the gradient of the obstacle function value corresponding to each agent. The forward safety constraint model is used to control each agent to always be within the safe range.

[0134] In this step, the forward safety constraint model is the dynamic differential constraint condition of the classical CBF mentioned above, which will not be elaborated further here.

[0135] Step S104: Input the training sample set into the task controller and output the initial action sequence; then, under the forward safety constraint model, correct the action sequence through quadratic programming to obtain the central safety action sequence of each agent.

[0136] In this step, a quadratic programming model is constructed by combining the control barrier function network, as follows:

[0137] (9);

[0138] in, The action sequence generated by the task controller based on the training set of samples currently being trained; The target action sequence to be optimized is defined by the same parameters as above, which will not be repeated here.

[0139] Under the premise of satisfying forward safety constraints, getting closer Ultimately, the desired control input with safety guarantees is obtained. Also This refers to a centralized safety action sequence, which is used to control all agents to both complete the target task and remain within a safe range.

[0140] Step S105: The central safety action sequence is used as a supervision label and input into the pre-constructed distributed control network along with the training sample set. The distributed action sequence of each agent is output, and the network parameters of the distributed control network are updated based on the pre-constructed second loss function. The distributed action sequence is used to control the corresponding agent to stay within the safety range.

[0141] In this embodiment of the invention, the distributed control network adopts an end-to-end learnable architecture based on graph neural networks, aiming to generate safe and cooperative control actions for each agent in a heterogeneous multi-agent system. The network constructs local subgraphs centered on each agent as input, and through node feature aggregation, edge information modeling, and attention mechanism fusion, ultimately outputs its corresponding distributed action sequence.

[0142] like Figure 4 As shown, the network structure of the distributed control network mainly includes an input module, an attention module, and a splicing module.

[0143] In one feasible implementation, the trajectory expansion state data of the agent and its neighbors are input to the distributed control network in the form of a heterogeneous local graph, which includes a center node, neighbor nodes, and edges; the corresponding information includes center node features, neighbor node features, and edge features. Based on this network structure, the process of the distributed control network outputting the distributed action sequence of each agent includes:

[0144] Step S105A: For each heterogeneous local graph, linearly map the center node features and neighbor node features thereof respectively to obtain initial embedding representations of the nodes.

[0145] In this step, the heterogeneous local subgraph information centered on the current agent is received by the input module, including: center node features (such as position, velocity, type), neighbor node features (state of other agents), and edge features (such as relative distance, relative velocity, safety radius, etc.).

[0146] Through linear mapping, the center node features and neighbor node features are linearly transformed respectively to extract initial embedding representations, so as to realize unified coding of the state space and support compatibility between heterogeneous agents.

[0147] Step S105B: Based on the initial embedding representations of the nodes, the attention weights of the neighbor node features relative to the center node features are calculated through a node attention mechanism; and the neighbor node features are weighted and fused based on the attention weights of the neighbor node features to obtain advanced features of the neighbor nodes.

[0148] The attention module includes node attention and edge attention. In this step, the importance weights of the neighbor nodes relative to the center node are calculated through a node attention mechanism. Specifically, the initial embedding representation of the center node is taken as a query, and the initial embedding of the neighbor node is taken as a key, and the attention weights of each neighbor to the current safety state are dynamically evaluated through the node attention mechanism.

[0149] Based on the attention weights of the neighbor nodes, the neighbor node features after linear mapping are weighted and fused to obtain advanced features of the neighbor nodes.

[0150] Step S105C: The edge attention weights of the edge features are calculated through an edge attention mechanism, and the edge features are weighted and fused based on the edge attention weights to obtain edge advanced features.

[0151] While linearly mapping the center node and the neighbor nodes, a multilayer perceptron is used to nonlinearly encode the edge features to extract advanced edge information (such as interaction intensity and potential conflict risk), so as to enhance the expression ability of complex interaction relationships.

[0152] Then the weight of the edge feature in message passing is calculated to give different priorities to different interaction relationships (such as obstacle avoidance and cooperation). And the weighted sum is performed based on the edge weight and the advanced edge information to obtain the advanced edge feature.

[0153] Step S105D: The advanced features of the neighbor nodes, the edge advanced features and the center node features are spliced to form a joint feature.

[0154] Step S105E: The joint feature is input into a multi-layer perception machine to output a distributed action sequence of each agent.

[0155] The joint feature is mapped to a final control action (such as speed and acceleration) through the last multi-layer perception machine; and a distributed action sequence with task guidance and safety is output.

[0156] The distributed control network in the embodiment of the application realizes accurate modeling of complex interaction relationships in a heterogeneous multi-agent system by introducing a node and edge double attention mechanism; the structural design ensures the integrity of information aggregation, and has good scalability and robustness, and can effectively generate special control actions suitable for different agent types, thereby supporting safe cooperative control of large-scale heterogeneous systems.

[0157] In a feasible implementation manner, updating the network parameters of the distributed control network based on the pre-constructed second loss function comprises:

[0158] Step A: determining an individual loss value of each agent according to the deviation between the distributed action sequence of each agent and the corresponding centralized safety action sequence.

[0159] The centralized safety action sequence is obtained by quadratic programming and is optimized based on globality, so as to ensure that no collision occurs between any two agents.

[0160] The distributed action sequence can only ensure that the agent itself does not collide with its neighbors, and cannot ensure that no collision occurs between other agents.

[0161] Therefore, the training target of the distributed control network is to make the distributed action sequence learn the centralized safety action sequence, so that the distributed control network can output an action sequence ensuring global optimization even based on local input.

[0162] For example, the individual loss function of the agent is as follows:

[0163] (10);

[0164] Wherein, represents the individual loss value of the agent . representing the agent a distributed action sequence; representing the agent a centralized safety action sequence.

[0165] Step B: the individual loss values of each agent are weighted and fused to obtain a total loss value of the second loss function.

[0166] In this step, different weights can be assigned to the individual loss values of each agent according to the agent type, task priority, or neighborhood size, etc.

[0167] For example, the second loss function is as follows:

[0168] (11);

[0169] wherein, is the weight of the individual loss value of the agent , and is the number of agents in the multi-agent collaboration system.

[0170] Step C: based on the total loss value, the network parameters of the distributed control network are updated through gradient backpropagation.

[0171] Step S106: repeat the above steps until the preset iteration termination condition is met.

[0172] In one example, the preset iteration termination condition is, for example, reaching a preset number of iterations or reaching a total loss value converging to a minimum.

[0173] With respect to the limitation of manually constructing the analytical form of the barrier function in the traditional CBF, the embodiment of the present application proposes an end-to-end learnable hGCBF network based on a graph neural network, which can automatically learn a barrier function suitable for heterogeneous multi-agents from multi-agent interaction data; on this basis, by embedding the safety constraints output by the hGCBF into a quadratic programming solver to generate a centralized safety action, and using this as a supervision signal to train a lightweight distributed control network, each agent can still output a high-reliability safety control instruction under the condition of only relying on local observations; not only overcoming the bottleneck of manually designing barrier functions, but also significantly improving the safety, robustness and task completion efficiency of the agents in executing actions in complex dynamic environments.

[0174] On the basis of the above embodiment, the method further comprises:

[0175] Step S107: based on the distributed action sequence output by the distributed control network, updating the initial trajectory state data of each agent and its neighbors.

[0176] Step S108: Dimensionally extend the updated trajectory state data and store it in the sample pool in the form of a graph structure.

[0177] The process of dimensionally extending and constructing the graph structure is described above and will not be repeated here.

[0178] Embodiments of the present application use the new action sequence generated by the distributed control network after each network parameter update to drive the agent to evolve new trajectory state data, which is stored in the sample pool in the form of a graph structure, thereby realizing the organic integration and dynamic evolution of new and old data:

[0179] On the one hand, the old data retains the exploration experience of the early strategy in diversified scenarios (including dangerous boundary areas), providing a wide generalization basis for the model.

[0180] On the other hand, the newly generated data closely reflects the behavior characteristics and safety performance of the current strategy, ensuring that the training signal always remains consistent with the latest strategy, effectively alleviating the distribution shift problem caused by policy iteration.

[0181] This new and old data cooperation mechanism not only improves the diversity and timeliness of the samples, but also enhances the stability and convergence speed of the joint training of the hGCBF network and the distributed control network, allowing the system to retain historical safety knowledge while continuously optimizing control performance, ultimately achieving more robust and efficient heterogeneous multi-agent safety cooperative control.

[0182] To more clearly illustrate the process of joint training of the hGCBF network and the distributed control network, Figure 5 a training framework diagram of the closed-loop iteration of the joint training of the two is given;

[0183] According to the framework, first, the hGCBF network is trained using the historical multi-agent state data as input, with the boundary conditions (i.e., the static classification of the CBF) and the differential constraints (i.e., the dynamic differential constraints of the CBF) as the supervision signals, and the parameters are updated; then, the safety constraints output by the hGCBF network are embedded in a quadratic programming problem to generate a centralized safety solution;

[0184] The distributed control network updates the parameters based on the difference between the output distributed solution and the centralized safety solution.

[0185] In this process, to reflect the differences in agent types, type weights are introduced to adjust the loss contribution of different agents, thereby guiding the distributed control network to learn and generate specialized control strategies that adapt to their respective characteristics.

[0186] After training, the distributed solution output by the distributed control network is used to drive the agent to perform actions and update its trajectory state data for subsequent training.

[0187] Based on the same inventive concept, an agent safety control method is provided, which is applied to a multi-agent cooperation system composed of at least two types of heterogeneous agents. Figure 6 As shown in the figure, the method comprises the following steps:

[0188] Step S601: Real-time acquisition of trajectory expansion state data of the target agent and its neighbors; the neighbors are other agents observed by each agent within a preset observation range.

[0189] Step S602: Inputting the trajectory expansion state data into the trained distributed control network to output a distributed action sequence of the target agent.

[0190] Based on the same technical concept, the embodiments of the present application also provide an electronic device, as shown in FIG. 7, which comprises a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702 and the memory 703 complete mutual communication through the communication bus 704.

[0191] The memory 703 is used for storing a computer program.

[0192] The processor 701 is used for executing the program stored in the memory 703, so as to realize the steps of the agent safety control network training method or the agent safety control method.

[0193] The communication bus mentioned in the above electronic device can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0194] The communication interface is used for communication between the above electronic device and other devices.

[0195] The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0196] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0197] In yet another embodiment provided by the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the method embodiments. For details, refer to the method embodiments, which will not be described here again.

[0198] Finally, it should be noted that: the above-described embodiments are merely specific implementations of the present application, which are used to illustrate the technical solutions of the present application, rather than limit the same. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easy changes to the technical solutions recorded in the foregoing embodiments within the technical scope disclosed by the present application, or can easily think of changes or equivalent replacements for some technical features. These modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training an intelligent agent security control network, characterized in that, The method is applied to a multi-agent cooperative system composed of at least two heterogeneous agents; the method includes: Obtain a training sample set, wherein each group of training samples in the training sample set contains trajectory extension state data of each agent and its neighbors at a certain sampling time. The training sample set is input into a pre-constructed heterogeneous graph control barrier function network, the barrier function values ​​corresponding to each agent are output, and the parameters of the heterogeneous graph control barrier function network are updated based on a preset first loss function. A forward safety constraint model is constructed based on the obstacle function value corresponding to each agent and the gradient of the obstacle function value. The forward safety constraint model is used to control each agent to always be within a safe range. The training sample set is input into the task controller, which outputs an initial action sequence. Under the forward safety constraint model, the initial action sequence is corrected by quadratic programming to obtain a central safety action sequence for each agent. The central safety action sequence is used to control all agents to complete the target task while remaining within a safe range. The central safety action sequence is used as a supervision label and input into a pre-constructed distributed control network along with the training sample set. The network output is the distributed action sequence of each agent, and the network parameters of the distributed control network are updated based on the pre-constructed second loss function. The distributed action sequence is used to control the corresponding agent to stay within a safe range. Repeat the above steps until the preset iteration termination condition is met.

2. The method according to claim 1, characterized in that, The acquisition of the training sample set includes: Under a preset operating strategy, the multi-agent cooperative system is run, and the initial trajectory state data of each agent and its neighbors are collected according to a preset sampling period; the neighbors are other agents observed by each agent within a preset observation range. The initial trajectory state data of each agent and its neighbors are expanded in dimension to obtain trajectory expanded state data, so as to unify the state dimension of heterogeneous agents. The trajectory extended state data of each agent and its neighbors are stored in the sample pool in the form of a graph structure; the node features of the graph structure include the type of agent and the safety radius; the edge features of the graph structure include the relative state and relative safety radius between each agent and its neighbors. A predetermined number of training samples are randomly selected from the sample pool periodically to form a training sample set.

3. The method according to claim 2, characterized in that, The method further includes: Based on the distributed action sequence output by the distributed control network, update the initial trajectory state data of each agent and its neighbors; The updated trajectory state data is dimensionally expanded and stored in the sample pool in the form of a graph structure.

4. The method according to claim 1, characterized in that, The first loss function is: in, This represents a preset safety margin; Extend state data for the trajectories of each agent and its neighbors; The set of security scopes for each intelligent agent; This is the output value of the barrier function; The gradient of the barrier function value; For extended classes function.

5. The method according to claim 1, characterized in that, The forward safety constraint model is as follows: in, This is the output value of the barrier function; The gradient of the barrier function value; For extended classes function.

6. The method according to claim 1, characterized in that, The trajectory extension state data of each agent and its neighbors are input into the distributed control network in the form of a heterogeneous local graph. The heterogeneous local graph includes a central node, neighboring nodes and edges, and the corresponding information includes central node features, neighboring node features and edge features. The process by which a distributed control network outputs the distributed action sequence of each agent includes: For each of the heterogeneous local graphs, a linear mapping is performed on the features of its central node and the features of its neighboring nodes to obtain the initial embedding representation of each node; Based on the initial embedding representation of each node, the attention weights of each neighbor node feature relative to the central node feature are calculated through a node attention mechanism; and the neighbor node features are weighted and fused based on the attention weights of each neighbor node feature to obtain the high-level features of the neighbor nodes. The edge attention weights of each edge feature are calculated using an edge attention mechanism, and the edge features are then weighted and fused based on these edge attention weights to obtain high-level edge features. The high-level features of the neighboring nodes, the high-level features of the edges, and the features of the center node are concatenated to form a joint feature; The joint features are input into a multilayer perceptron, which outputs a distributed action sequence for each agent.

7. The method according to claim 6, characterized in that, Updating the network parameters of the distributed control network based on the pre-constructed second loss function includes: The individual loss value of each agent is determined based on the deviation between the distributed action sequence of each agent and the corresponding centralized safety action sequence. The individual loss values ​​of each agent are weighted and fused to obtain the total loss value of the second loss function; Based on the total loss value, the network parameters of the distributed control network are updated through gradient backpropagation.

8. A method for intelligent agent safety control, characterized in that, The method is applied to a multi-agent cooperative system composed of at least two heterogeneous agents; the method includes: The trajectory extension state data of the target intelligent agent and its neighbors are acquired in real time; the neighbors are other intelligent agents observed by each intelligent agent within a preset observation range. The trajectory extension state data is input into the distributed control network trained based on the method described in any one of claims 1-7, and the distributed action sequence of the target agent is output.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images