Multi-agent cooperative security control system and method based on adaptive game architecture

Abstract

The application discloses a multi-agent collaborative security control system and method based on an adaptive game architecture, relates to the field of robot control, and comprises a multi-player decision module that generates an action recommendation vector; an adaptive mixer module that dynamically calculates the fusion weight of each action recommendation based on an attention mechanism, performs weighted summation and is processed by a differentiable projection function, and outputs a joint action that meets the boundary of the action space; a risk perception evaluation module that outputs the estimated value of the cumulative return and the cumulative cost at the preset quantile point, and extracts a risk-sensitive index; and a soft scheduler module that generates a satisfaction signal according to the comparison result of the cumulative cost of each constraint and the preset safety threshold, and converts the satisfaction signal into a time-varying learning rate coefficient through a smoothing mapping function to modulate the update step, thereby realizing the high state adaptability of action synthesis, the smooth stability of policy updating, and the precise control of extreme security risks, and significantly improving the task completion rate and safety of the robot in a complex and restricted environment.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a multi-agent collaborative safety control system and method based on an adaptive game theory architecture. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep reinforcement learning has demonstrated remarkable adaptability in safety-critical fields such as robot control, autonomous driving, and industrial automation. However, how to provide system-level safety redundancy while ensuring optimal task performance has become a bottleneck restricting its large-scale engineering implementation. Currently, the research field generally uses constrained Markov decision processes to model such constrained optimization problems. Existing technologies mainly solve these problems through the Lagrange dual method and the trust domain policy optimization method. The former is highly dependent on the fine adjustment of the dual variables and is prone to gradient conflicts that lead to training oscillations. The latter involves high-dimensional matrix inverse operations, has huge computational overhead, and is difficult to adapt to high-dimensional state spaces or environments with extremely high real-time response requirements. While existing game theory-based constrained reinforcement learning frameworks have significant advantages in mitigating gradient conflicts, they still have obvious shortcomings in real-world complex dynamic environments: First, the rigidity of joint policy synthesis. Existing methods mostly use static action weighting to synthesize the final action, and their weight allocation logic is decoupled from system state awareness, making it impossible to dynamically adjust the adoption ratio of each player's suggestions according to the real-time risk situation. Second, the instability of the update mechanism. The dominant time scale update criterion based on hard switching is prone to triggering instantaneous changes in dominance under constrained critical states, causing violent oscillations in policy parameters and impairing the stability and robustness of the learning process. Third, the limitations of risk perception capabilities. Mainstream frameworks mainly rely on expected value for risk assessment and policy guidance, lacking effective characterization of distributed tail risks, resulting in insufficient protection of agents against low-probability, high-harm long-tail disaster risks. Summary of the Invention

[0003] To address the aforementioned challenges, this application provides a multi-agent collaborative safety control system and method based on an adaptive game theory architecture. Through a collaborative mechanism of adaptive hybrid action synthesis, soft scheduling smooth updates, and distributed risk perception and evaluation, a dynamic balance between task performance and safety constraints in complex and constrained environments is achieved.

[0004] To achieve the above objectives, this application provides a multi-agent cooperative safety control system based on an adaptive game theory architecture, comprising: The multi-player decision-making module includes a task agent equipped with a task policy network and multiple constraint agents equipped with independent constraint policy networks, used to generate action suggestion vectors; The adaptive mixer module, connected to the multiplayer decision module, is used to dynamically calculate the fusion weights of each action proposal based on the attention mechanism, perform weighted summation and process it through a differentiable projection function, and output a joint action that conforms to the action space boundary. The risk perception and evaluation module, connected to the adaptive mixer module, includes a task value subnetwork and multiple constraint cost subnetworks using a quantile regression architecture; it is used to output the estimated values ​​of cumulative return and cumulative cost at preset quantiles and extract risk-sensitive indicators. The soft scheduler module, connected to the multi-player decision-making module and the risk perception evaluation module, is used to generate a satisfaction signal based on the comparison results of the cumulative cost of each constraint and the preset safety threshold. The signal is then converted into a time-varying learning rate coefficient through a smoothing mapping function to modulate the update step size of each policy network.

[0005] Preferably, the adaptive mixer module specifically includes: The attention weight calculation unit is used to extract global state features and intermediate layer features of each policy network, and calculates the normalized weights through the scaling dot product attention mechanism. The nonlinear weighted fusion unit is used to perform weighted summation and ensure the physical feasibility of the output action through a differentiable projection function; The risk-adaptive gating unit is used to dynamically adjust the fusion weights based on risk-sensitive indicators.

[0006] Preferably, the risk perception and assessment module includes: The task value subnetwork is used to estimate the cumulative return at a preset quantile. Multiple constraint cost subnetworks are used to estimate the estimated values ​​of each security constraint cost at preset quantiles; The risk indicator extraction unit is used to extract risk-sensitive indicators based on the estimated values ​​of the task value subnetwork and multiple constrained cost subnetworks.

[0007] Preferably, the soft scheduler module includes: The satisfaction monitoring unit is used to monitor the cumulative cost of each constraint in real time and compare it with a preset safety threshold to generate a continuous satisfaction signal. The smoothing mapping unit is used to map the satisfaction signal into time-varying learning rate coefficients through a parameterized function. The update scheduling unit is used to allocate gradient update dominance between the task agent and each constraint agent according to the learning rate coefficient.

[0008] A multi-agent cooperative safety control method based on an adaptive game theory architecture, applied to a multi-agent cooperative safety control system based on an adaptive game theory architecture, includes the following steps: S1: Construct a distributed decision architecture based on Markov game theory, decouple the controlled system into a task agent and multiple constraint agents. The task agent is equipped with a task policy network, and each constraint agent is equipped with an independent constraint policy network. The weight parameters of the task policy network and the constraint policy network are randomly initialized. S2: At each decision moment, the environmental state observation vector is simultaneously input into the task policy network and the constraint policy network to obtain the action suggestion vector; the fusion weight of each action suggestion is dynamically calculated based on the attention mechanism, weighted summation is performed and processed by a differentiable projection function, and the joint action that conforms to the action space boundary is output. After the joint action interacts with the environment, the interaction data is stored in the experience replay pool. S3: Sample batch data from the experience replay pool and input it into the risk perception evaluation module; the risk perception evaluation module contains a parallel task value sub-network and multiple constraint cost sub-networks, all of which adopt a quantile regression architecture and output the estimated values ​​of cumulative return and cumulative cost at preset quantiles respectively; train the risk perception evaluation module based on the quantile loss function and extract risk-sensitive indicators; S4: Generate a satisfaction signal based on the comparison between the cumulative cost of each constraint and the preset safety threshold, and convert it into a time-varying learning rate coefficient through a smoothing mapping function; S5: Based on the risk-sensitive index and the time-varying learning rate coefficient, update the task policy network and each constraint policy network, wherein the update step size of the task policy network and each constraint policy network is modulated by the corresponding learning rate coefficient.

[0009] Preferably, the method for dynamically calculating the fusion weights based on the attention mechanism in S2 includes: Extract global state features and features of intermediate layers of each policy network; The correlation score between each policy feature and the global state feature is calculated using a scaled dot product attention mechanism. The correlation scores are normalized into fusion weights using the Softmax function.

[0010] Preferably, S2 also includes a risk adaptive gating step, specifically including: When the risk sensitivity indicators extracted by the risk perception and evaluation module exceed the preset risk threshold, the calculation process of the attention mechanism is dynamically adjusted to dynamically increase the fusion weight corresponding to the safety constraint agent.

[0011] Preferably, the quantile loss function in S3 is the quantile Huber loss function, expressed as: ; in, The target quantile is calculated based on the target network and the Bellman equation. The quantile Huber loss function, It is the preset total number of quantiles. For state-action pairs Find the expected value. The output of the task value subnetwork at the quantile The estimated value.

[0012] Preferably, the soft scheduler generates a satisfaction signal based on the comparison between the cumulative cost of each constraint and a preset safety threshold, specifically including: Real-time monitoring of the cumulative cost of each constrained agent; Calculate the difference between the accumulated cost and the preset safety threshold; By using a buffered clipping function, the gap is mapped to a satisfaction signal within the interval.

[0013] Preferably, the smoothing mapping function is a parameterized Sigmoid-like function that nonlinearly maps the satisfaction signal into time-varying learning rate coefficients.

[0014] Therefore, the multi-agent cooperative security control system and method based on an adaptive game theory architecture described above have the following beneficial effects: (1) This application achieves state adaptability in joint action synthesis through the design of an adaptive mixer, solving the problems of rigid strategy fusion logic and coarse weight adjustment in the prior art. By using an attention mechanism to dynamically weight the action suggestions of each player, the system can automatically reconstruct the adoption ratio of each player's suggestions based on the real-time environmental risk situation (such as obstacle distance and energy reserves). This nonlinear mapping mechanism ensures that the agent can finely balance task performance and safety boundaries under complex conditions, significantly improving the collaborative efficiency of the system in constrained dynamic environments.

[0015] (2) This application adopts a smooth update strategy based on a soft scheduler, which effectively overcomes the training instability caused by the "hard switching" mechanism in the traditional GCPO. By introducing a continuous modulation function based on the satisfaction signal, this application achieves a smooth transition of dominance between the task player and the constraint player. This stable learning mechanism avoids drastic jumps and oscillations of policy parameters near the constraint boundary, which not only significantly accelerates the convergence speed of the algorithm, but also ensures the smoothness of the final execution action by maintaining the continuity of policy updates, greatly improving the physical stability of the executor in the dynamic interaction process.

[0016] (3) In terms of risk control, this application introduces a distributed risk perception and evaluation system, which fills the gap in the modeling of extreme risks (long-tail risks) in existing solutions. By learning the complete probability distribution of returns and costs through quantile regression technology and extracting conditional value at risk (CVaR) as a guiding indicator, the system can explicitly characterize and optimize catastrophic consequences that have a low probability of occurrence but are extremely harmful. Compared with traditional solutions that rely solely on expected value estimation, this application can identify and intercept potential extreme risks in advance when facing highly uncertain dynamic environments, providing a more stringent and reliable security guarantee for autonomous decision-making in safety-critical areas.

[0017] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall architecture of a multi-agent collaborative safety control system based on an adaptive game theory architecture, as described in this application. Figure 2 This is a schematic diagram of the decomposition action results in the embodiments of this application; Figure 3 This is a schematic diagram of the learning process trajectory in the embodiments of this application; Figure 4 This is a schematic diagram of the heat map comparison results in the embodiments of this application. Detailed Implementation

[0019] The following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0020] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning as understood by a person of ordinary skill in the art to which this application pertains.

[0021] The terms "comprising" or "including," as used in this application, mean that the element preceding the term encompasses the element listed after the term, and do not exclude the possibility of encompassing other elements as well. The terms "inner," "outer," "upper," and "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In this application, unless otherwise expressly specified and limited, the term "attached," etc., should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can refer to a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication of two elements or the interaction relationship between two elements. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0022] Example 1: A multi-agent collaborative safety control system based on an adaptive game theory architecture adopts a hierarchical decoupling and modular collaborative design concept. Functionally, it can be divided into four core layers: perception and decision-making layer, action fusion layer, risk assessment layer, and learning and scheduling layer. Each layer achieves collaborative optimization through closed-loop data flow and feedback mechanisms. Figure 1 As shown, it includes: The multi-player decision-making module includes a task agent equipped with a task policy network and multiple constraint agents equipped with independent constraint policy networks, used to generate action suggestion vectors; Specifically, the multi-player decision-making module is a distributed decision-making architecture based on general and Markov games. This module decouples the multi-objective security control problem by dividing it into several independent decision-making players with specific optimization orientations. This effectively separates task benefits from security constraints at the logical level, mitigating the decision instability of traditional frameworks when dealing with conflicting objectives.

[0023] The task decision unit is equipped with a policy network. Based on environmental observation Output is designed to maximize cumulative rewards Action suggestions The parallel constraint decision unit group contains Each unit is an independent unit. All are configured with policy networks The output for a specific security dimension aims to minimize the cumulative constraint cost. Action suggestions This distributed unit architecture ensures that the system can perform fine-grained monitoring and real-time response to multi-dimensional physical boundaries.

[0024] The system's policy networks all employ deep neural networks, performing random parameter configuration during initialization and iterative optimization through a parallel update mechanism during training. Through the collaborative evolution of multiple sub-policies within a game-theoretic framework, the system can continuously generate high-quality candidate action sets, laying the foundation for subsequent adaptive action mixing. This design ensures the system's ability to handle high-dimensional state spaces in complex and constrained environments, achieving a dynamic balance between security and performance.

[0025] The adaptive mixer module, connected to the multiplayer decision module, is used to dynamically calculate the fusion weights of each action proposal based on the attention mechanism, perform weighted summation and process it through a differentiable projection function, and output a joint action that conforms to the action space boundary. The adaptive mixer module specifically includes: The attention weight calculation unit is used to extract global state features and intermediate layer features of each policy network, and calculates the normalized weights through the scaling dot product attention mechanism. The nonlinear weighted fusion unit is used to perform weighted summation and ensure the physical feasibility of the output action through a differentiable projection function; The risk-adaptive gating unit is used to dynamically adjust the fusion weights based on risk-sensitive indicators.

[0026] Specifically, the adaptive mixer module, as the core mixing layer of the system, connects to the multi-player decision-making module through a logical interface and is responsible for fusing action weights based on state awareness. Internally, it integrates an attention weight calculation unit, simultaneously extracting global state representations and intermediate layer features from each policy network, and using a scaled dot product attention mechanism to calculate normalized weights. This state-driven logic ensures that the system can dynamically and precisely identify the decision contribution of each sub-policy based on real-time conditions.

[0027] At the action synthesis level, the nonlinear weighted fusion unit performs weighted summation and ensures the physical feasibility of the output action through a differentiable projection function. This design ensures the continuity of gradient flow while forcing the joint action to strictly stay within the system action space boundary, effectively avoiding the risk of actuators going out of bounds at the architectural level.

[0028] Furthermore, this module deeply integrates a risk-adaptive gating mechanism, receiving risk signals from the evaluation layer in real time. When risk indicators such as cost distribution quantiles exceed preset thresholds, the gating mechanism dynamically intervenes in the weight calculation unit, significantly increasing the weight ratio of players subject to safety constraints, thereby rapidly shifting the strategy focus towards the safety dimension. This responsive architecture endows the system with the closed-loop control capability to prioritize the safety boundary when facing extreme negative risks.

[0029] The risk perception and evaluation module, connected to the adaptive mixer module, includes a task value subnetwork and multiple constraint cost subnetworks using a quantile regression architecture; it is used to output the estimated values ​​of cumulative return and cumulative cost at preset quantiles and extract risk-sensitive indicators. The risk perception and assessment module includes: The task value subnetwork is used to estimate the cumulative return at a preset quantile. Multiple constraint cost subnetworks are used to estimate the estimated values ​​of each security constraint cost at preset quantiles; The risk indicator extraction unit is used to extract risk-sensitive indicators based on the estimated values ​​of the task value subnetwork and multiple constrained cost subnetworks.

[0030] Specifically, the risk perception and evaluation module, as the system's evaluation layer, provides distributed risk assessment capabilities for decision-making and hybrid processes that go beyond traditional expected value dimensions. This module employs a distributed quantile regression network architecture, which, through refined modeling of the full probability distribution of the value function, can quantify and capture uncertain risks in the environment. Its internally integrated task value sub-network is responsible for estimating cumulative returns. The estimated values ​​at preset quantiles fully characterize the shape of the reward distribution, providing high-dimensional evaluation guidance for improving task performance.

[0031] The constrained cost subnetwork group constructed in parallel includes Each independent evaluation unit addresses various safety constraint costs. Quantile distribution estimation is performed. Based on this, the module's built-in risk indicator extraction unit can calculate core tail risk indicators such as conditional value of risk based on the quantile output. The risk perception signal generated by this unit not only reflects the average risk under the current state, but also explicitly reveals the low-probability, high-hazard long-tail risk situation.

[0032] The extracted risk signals are transmitted in real time to the adaptive mixer module and the soft scheduler module via a feedback link, enabling a dynamic shift of the system's decision focus towards higher-risk dimensions. The evaluation module as a whole undergoes iterative training using the quantile Huber loss function, aiming to accurately capture and model the tail features of the value distribution. This distributed evaluation architecture ensures that the system possesses robust control capabilities to anticipate and avoid extreme accidents under dynamic, uncertain, and complex operating conditions, providing solid risk quantification support for multi-objective safety collaboration.

[0033] The soft scheduler module, connected to the multi-player decision-making module and the risk perception evaluation module, is used to generate a satisfaction signal based on the comparison results of the cumulative cost of each constraint and the preset safety threshold. The signal is then converted into a time-varying learning rate coefficient through a smoothing mapping function to modulate the update step size of each policy network.

[0034] The soft scheduler module includes: The satisfaction monitoring unit is used to monitor the cumulative cost of each constraint in real time and compare it with a preset safety threshold to generate a continuous satisfaction signal. The smoothing mapping unit is used to map the satisfaction signal into time-varying learning rate coefficients through a parameterized function. The update scheduling unit is used to allocate gradient update dominance between the task agent and each constraint agent according to the learning rate coefficient.

[0035] Specifically, the soft scheduler module, as part of the system's scheduling layer architecture, has the core function of ensuring a smooth transition of leadership during policy updates, aiming to eliminate training oscillations caused by traditional hard switching mechanisms. The satisfaction monitoring unit integrated within this module can capture the cumulative cost feedback from each constraint decision unit in real time. and compare it with a preset safety threshold. High-frequency comparison is performed. By calculating the instantaneous difference between the two, the unit generates a value that is in a certain position. Continuous satisfaction signal of interval This quantifies the abstract constraint satisfaction state into a numerical representation that can be used for subsequent computation and scheduling. This continuous monitoring mechanism provides the system with accurate feedback on the safety boundary margin.

[0036] The module's built-in smoothing mapping unit uses a parameterized Sigmoid-like function to perform a nonlinear transformation on the aforementioned satisfaction signal, mapping it to time-varying learning rate coefficients. This mapping process achieves fine-grained intervention in the gradient descent trajectory by dynamically modulating the update step size of each policy network. When the system detects an increase in the risk of constraint violation, the mapping function automatically adjusts its output, significantly increasing the update strength of the corresponding constraint policy; conversely, it gently relinquishes dominance. This scheduling logic based on continuous functions fundamentally changes the "either / or" update pattern between game players, ensuring the continuity of parameter space optimization.

[0037] The update scheduler of the update scheduling unit updates the coefficients. The system allocates dominance for gradient updates among the task player and the constrained players. This soft-modulation scheduling strategy allows control to be transferred gradually between different players, effectively avoiding drastic jumps in policy parameters during dominance transitions. This scheduling layer architecture not only significantly enhances the stability of policy iteration in complex and constrained environments but also ensures that the system can still produce high-performance and highly secure collaborative control schemes under highly dynamic conditions by maintaining the stability of the convergence path.

[0038] Example 2: A multi-agent cooperative safety control method based on an adaptive game theory architecture, applied to a multi-agent cooperative safety control system based on an adaptive game theory architecture, includes the following steps: S1: Construct a distributed decision architecture based on Markov game theory, decouple the controlled system into a task agent and multiple constraint agents. The task agent is equipped with a task policy network, and each constraint agent is equipped with an independent constraint policy network. The weight parameters of the task policy network and the constraint policy network are randomly initialized. Specifically, to achieve multi-objective safe collaborative control, this application first constructs a distributed decision-making architecture based on general and Markov games. This architecture decouples the controlled system into a task-oriented intelligent agent and... An independent decision-making unit consisting of several constrained intelligent agents. The task-oriented intelligent agent is equipped with a policy network. Responsible for receiving environmental observation vectors And output to maximize cumulative rewards Action suggestions for the target Each constrained agent Each is equipped with an independent policy network When receiving the same observations Based on this, the output aims to minimize the corresponding cumulative constraint cost. Action suggestions All policy networks employ a deep neural network structure, with their weight parameters randomly initialized at the start of training, laying the foundation for subsequent parallel game evolution.

[0039] In the decision synthesis dimension, the system is configured with an adaptive mixer module to achieve efficient fusion of action proposals from various agents. This module receives the global state representation and the action proposal vector sets from each agent. As input, its embedded attention weight calculation network can dynamically generate a set of normalized fusion weights based on the system's real-time risk situation and task requirements. Through this state-aware nonlinear mapping mechanism, the mixer can replace the traditional hard weighting, ensuring that the joint action can balance task performance and safety boundaries under different operating conditions.

[0040] To ensure learning stability during the game process, the system synchronously initializes a soft scheduler module to achieve smooth policy learning scheduling. This module continuously monitors the cost function values ​​of each constraint. With preset safety threshold The instantaneous difference between them is used to calculate the satisfaction signal of each agent in real time. The soft scheduler's built-in smoothing mapping functions (such as Sigmoid-like functions) will express the satisfaction signal. Transformed into time-varying learning rate coefficients This allows for dynamic adjustment of the update step size of each policy network during training, effectively avoiding parameter oscillations and performance collapse that may be caused by the instantaneous switching of dominance.

[0041] At the evaluation mechanism level, the system constructs a risk perception evaluation module, aiming to provide risk perception assessments that go beyond the expected value dimension. This network consists of multiple parallel quantile regression subnetworks, each used to estimate future cumulative returns. Cumulative costs across various dimensions The probability distribution is calculated. By outputting estimates of the value function at multiple preset quantiles, this risk perception and evaluation module can accurately characterize the tail features of the distribution, providing a quantitative evaluation basis for agents to cope with "long-tail" catastrophic risks. The parameters of this risk perception and evaluation module are initialized independently of the policy network and iteratively optimized using distributed reinforcement learning criteria.

[0042] S2: At each decision moment, the environmental state observation vector is simultaneously input into the task policy network and the constraint policy network to obtain the action suggestion vector; the fusion weight of each action suggestion is dynamically calculated based on the attention mechanism, weighted summation is performed and processed by a differentiable projection function, and the joint action that conforms to the action space boundary is output. After the joint action interacts with the environment, the interaction data is stored in the experience replay pool. Methods for dynamically calculating fusion weights based on attention mechanisms include: Extract global state features and features of intermediate layers of each policy network; The correlation score between each policy feature and the global state feature is calculated using a scaled dot product attention mechanism. The correlation scores are normalized into fusion weights using the Softmax function.

[0043] Specifically, at each decision-making moment The agent receives state observation vectors from the environment. The observation vector is simultaneously input to all initialized policy networks, including a task policy network. and A constrained policy network Each network independently performs forward inference, generating its corresponding action suggestion output, which together constitutes the action suggestion vector. .

[0044] Subsequently, the action vector is compared with the current state. The data is fed into an adaptive mixer. Inside the mixer, it is first processed through a feature extraction layer. To obtain high-level state representation At the same time, the mixer synchronously obtains the specific feature encodings of each policy network from a pre-defined intermediate layer. An attention scoring function based on scaled dot products is used to calculate the relevance score between each player feature and the global state representation.

[0045] Based on these scores, the mixer applies the Softmax function for normalization, generating a set of weight coefficients. The set of coefficients satisfies Its numerical value dynamically reflects the current state. The relative importance of the suggested actions for each player is determined. The mixer performs a weighted summation operation to calculate the initial joint actions. To ensure the physical feasibility of the output action, the preliminary result is processed by a differentiable projection function, ultimately generating and outputting a joint action that conforms to the system's action space boundary. .

[0046] This process also includes a risk-adaptive gating mechanism. Real-time risk assessment signals from risk-aware distributed commentators (e.g., high quantile estimates of the predicted cost distribution) are used as gating inputs. When this signal exceeds a preset safety threshold, the gating mechanism dynamically adjusts the attention score calculation process, thereby assigning a higher proportion of safety-constrained players to the weighting coefficients, achieving a rapid shift of the strategy focus towards the safety dimension.

[0047] The agent will ultimately generate joint actions The data is sent to the actuator to interact with the environment. Environmental feedback includes instantaneous rewards. Instantaneous costs of various constraints and the state observation at the next moment Complete interactive data tuples ( The data is systematically stored in the experience replay pool to provide data support for subsequent strategy evaluation and update phases.

[0048] S3: Sample batch data from the experience replay pool and input it into the risk perception evaluation module; the risk perception evaluation module contains a parallel task value sub-network and multiple constraint cost sub-networks, all of which adopt a quantile regression architecture and output the estimated values ​​of cumulative return and cumulative cost at preset quantiles respectively; train the risk perception evaluation module based on the quantile loss function and extract risk-sensitive indicators; Specifically, a batch of historical interaction data is sampled from the experience replay pool. For each data sample, the state is... Joint operations carried out at the time The inputs are fed into the risk perception and assessment module. This network contains parallel sub-network modules, where the task value sub-network receives the inputs and outputs the cumulative future returns. exist Preset quantiles On the estimated value This characterizes the probability distribution of returns. Meanwhile, Each constrained cost subnetwork outputs its corresponding cumulative cost. The set of estimates at the same quantile.

[0049] The training objective of the network is to minimize the distribution projection error. For a task-value network, its loss function is defined as the sum of the quantile Huber losses at each quantile: ; in, The target quantile is calculated based on the target network and the Bellman equation. The quantile Huber loss function, It is the preset total number of quantiles. For state-action pairs Find the expected value. The output of the task value subnetwork at the quantile The estimated value is given above. Each constrained cost subnetwork uses the same form of loss function. Conduct independent training.

[0050] Distributed commentators trained in this way are not only able to provide expected value estimates More importantly, it can depict the complete shape of value distribution. Based on the quantile estimation of the output, the system can directly calculate risk sensitivity indicators, such as conditional value at risk (CVaR). For the quantile... One constraint, at the confidence level The conditional value of risk can be estimated as follows: ; This indicator provides a quantitative perception and constraint basis for subsequent strategy optimization regarding extreme negative risks (i.e., tail risks).

[0051] S4: Generate a satisfaction signal based on the comparison between the cumulative cost of each constraint and the preset safety threshold, and convert it into a time-varying learning rate coefficient through a smoothing mapping function; The satisfaction signal is generated based on the comparison between the cumulative cost of each constraint and the preset safety threshold, specifically including: Real-time monitoring of the cumulative cost of each constrained agent; Calculate the difference between the accumulated cost and the preset safety threshold; By using a buffered clipping function, the gap is mapped to a satisfaction signal within the interval.

[0052] The smoothing mapping function is a parameterized Sigmoid-like function that nonlinearly maps the satisfaction signal into time-varying learning rate coefficients.

[0053] Specifically, based on the current strategy, the system evaluates the expected cumulative cost corresponding to each constraint. The soft scheduler will With preset threshold By comparison, a satisfaction signal within the interval [0,1] is generated using a buffered pruning function. This signal reflects the degree to which each constraint is satisfied.

[0054] Then, the scheduler will assign each Given a parameterized smooth sigmoid mapping function, calculate the continuous, time-varying learning rate coefficients. This mechanism ensures that when constraints are not met ( (smaller), corresponding to the player's Significantly increases, leading to a dominant policy update; when constraints are satisfied ( Larger), quest player With the corresponding improvement, the dominant position can be smoothly transferred.

[0055] S5: Based on the risk-sensitive index and the time-varying learning rate coefficient, update the task policy network and each constraint policy network, wherein the update step size of the task policy network and each constraint policy network is modulated by the corresponding learning rate coefficient.

[0056] System execution strategy update: The network parameters of the task player are updated to maximize expected return. Gradient direction update, step size changed from Modulation; each constrained player's network parameters are adjusted to minimize their perceived risk cost metric (such as conditional value of risk). The gradient direction is updated, and the step size is changed by the corresponding gradient direction. Modulation. All updates are performed synchronously, and because... Continuous changes effectively avoid policy oscillations caused by hard switching, ensuring the stability of the training process.

[0057] This application achieves high state adaptability in motion synthesis, smooth stability in policy updates, and precise control over extreme safety risks, significantly improving the robot's task completion rate and safety in complex and constrained environments.

[0058] Example 3: To more intuitively compare the performance differences between this application and existing technologies, we recorded the evolution of key metrics during typical training rounds, and the results are as follows: Figures 2-4 As shown.

[0059] like Figure 2The diagram illustrates the action decomposition at each step of a trajectory drawn on an environmental map. Red rectangles represent danger zones, green rectangles represent target zones, and blue dots mark the starting point. Three arrows are drawn along the trajectory (every few steps): green represents the action output by the task player; red represents the action constraining the player's output; and blue represents the total action. This diagram visually demonstrates the contributions of the agent's two sub-policies and the actual control quantity resulting from their synthesis. Figure 2 This diagram illustrates the relationship between the player's actions, constrained player actions, and the final composite action at each step point along a typical trajectory. The diagram shows that: (1) The actions output by the task player are generally directed towards the target area, reflecting that the task strategy network always takes improving the efficiency of reaching the target and shortening the path length as the main optimization direction; (2) The constraint on the player's output action shows a clear tendency to avoid danger when approaching the danger zone, indicating that the constraint policy network can generate safety-oriented corrective actions for danger zones, collision risks or boundary risks. (3) The final blue total action is not simply equal to a single sub-policy, but is a dynamic compromise direction between task action and constraint action, indicating that the adaptive mixer module in this application can adjust the fusion weight in real time according to the current state to achieve online balance between task benefits and security constraints. (4) When the trajectory is far away from the danger zone, the total action is closer to the task player's output; when the trajectory is close to the danger zone, the total action is closer to the constrained player's output. This shows that the attention fusion mechanism and the risk adaptive gating mechanism can dynamically change the control dominance according to the environmental risk situation and have significant state adaptability. Therefore, by Figure 2 It can be demonstrated that this application achieves an interpretable fusion of "task-driven" and "safety-driven" at the action level, which can intuitively reflect the functional division of each sub-strategy and the formation mechanism of the final joint action, thereby verifying the effectiveness of the adaptive action hybrid mechanism.

[0060] Figure 3 This demonstrates the overlay of multiple trajectory paths on an environmental map. Red rectangles represent danger zones, green rectangles represent target zones, and blue dots mark the starting points. Up to 500 trajectories are uniformly sampled from multiple episodes, and their paths are drawn with thin (semi-transparent) lines. Figure 3 This figure shows the distribution of a large number of trajectories obtained from downsampling over multiple training rounds. It can be seen from this figure that: (1) Most trajectories can stably reach the vicinity of the target area from the starting point, indicating that the method proposed in this application has high task completion ability and good convergence performance; (2) The trajectory generally shows obvious detour characteristics to dangerous areas. There are fewer trajectories inside and near the boundaries of dangerous areas, indicating that this application can effectively suppress the agent from entering high-risk areas and has good safety avoidance capabilities. (3) As training progresses, the trajectory distribution tends to be concentrated rather than randomly diverging, indicating that the soft scheduler module effectively reduces the policy oscillations caused by the traditional hard switching update method by continuously modulating the learning rate, making the training process more stable. (4) Multiple trajectories show good path consistency when approaching the target area, reflecting that the system can maintain high task execution stability and decision consistency while satisfying safety constraints. Therefore, by Figure 3 It can be demonstrated that this application can not only improve the success rate of missions, but also form a stable, safe and repeatable control strategy in multi-round training, and verify the effectiveness of the soft scheduling smooth update mechanism and the multi-agent collaborative decision-making mechanism from the perspective of trajectory distribution.

[0061] Figure 4 The heatmap comparison results are displayed, where the horizontal axis represents the environment x-direction (0-10) and the vertical axis represents the environment y-direction (0-10). A cool-warm color bar is used, where red / warm colors generally indicate a higher Q value (higher expected return) and blue / cool colors indicate a lower Q value (lower expected return). The specific numerical range is given by the color bar. Figure 4 A heatmap showing the value distribution in the environmental state space is presented. This map shows that: (1) The area near the target area usually shows a warm color high value distribution, indicating that the system can correctly identify the high return attribute corresponding to the target area and highlight the target orientation in value assessment; (2) The area near the danger zone usually shows a low value distribution of cold color, and there is an obvious value gradient change around the danger boundary. This indicates that the risk perception evaluation module of this application can effectively encode the potential cost of the danger zone into the value estimation, so that the system naturally tends to avoid high-risk states when making decisions. (3) When transitioning from a dangerous area to a safe passage area, the heat map color shows a continuous change rather than an abrupt break, indicating that the value learning process of this application is relatively smooth and the state value function has good continuity and interpretability, which is consistent with the smooth update effect brought by the soft scheduler. (4) High-value areas in the heat map usually extend gradually towards the target area along a safe and reachable path, indicating that the system learns not just the shortest path preference, but a "safe and high-yield path" that takes into account risk avoidance. Therefore, by Figure 4It can be demonstrated that this application, through quantile regression and risk-sensitive index extraction, can form a state value distribution representation that takes into account both returns and security costs, thus verifying the effectiveness of the distributed risk perception evaluation mechanism for tail risk modeling and security decision guidance from the perspective of value assessment.

[0062] according to Figures 2-4 It is known that this application significantly improves the task completion rate and operational safety of multi-agent systems in complex and constrained environments through the synergistic effect of three core mechanisms: adaptive action mixing, soft scheduling smooth update, and distributed risk perception evaluation. It overcomes key technical challenges in existing technologies, such as rigid strategy synthesis, unstable update mechanism, and limited risk perception capabilities. Figure 2 From the perspective of action synthesis, this application can achieve a dynamic trade-off between task-oriented and safety-oriented approaches based on the real-time risk status. Figure 3 From the perspective of trajectory evolution, this application demonstrates that it can form a stable, convergent, and safe control path during multi-round training. Figure 4 From the perspective of value distribution, this application can accurately identify target benefits and risky costs, and incorporate tail risk information into the value assessment process.

[0063] This further illustrates that the adaptive action hybrid mechanism, soft scheduling smooth update mechanism, and distributed risk perception and evaluation mechanism proposed in this application form an effective synergy: the former is responsible for realizing real-time action-level collaborative decision-making, the middle is responsible for ensuring the stability of the training and update process, and the latter is responsible for providing a forward-looking quantitative assessment of extreme risks. The combined effect of these three mechanisms enables this application to simultaneously achieve a high task completion rate, strong security assurance capabilities, and good strategy interpretability in complex and constrained environments.

[0064] Therefore, this application adopts the aforementioned multi-agent collaborative safety control system and method based on an adaptive game theory architecture. It uses an attention mechanism to dynamically allocate the fusion weights of tasks and safety actions to achieve collaborative decision-making with state awareness; it continuously modulates the learning rate of each player based on the satisfaction signal to achieve a smooth transition of dominance and avoid policy oscillation; it models the value distribution through quantile regression to quantify and control tail risk with conditional risk value; this application integrates the above mechanisms to form a general and scalable safety reinforcement learning architecture, which is suitable for single / multi-agent collaborative control.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of this application, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of this application.

Claims

1. A multi-agent cooperative safety control system based on an adaptive game theory architecture, characterized in that, include: The multi-player decision-making module includes a task agent equipped with a task policy network and multiple constraint agents equipped with independent constraint policy networks, used to generate action suggestion vectors; The adaptive mixer module, connected to the multiplayer decision module, is used to dynamically calculate the fusion weights of each action proposal based on the attention mechanism, perform weighted summation and process it through a differentiable projection function, and output a joint action that conforms to the action space boundary. The risk perception and evaluation module, connected to the adaptive mixer module, includes a task value subnetwork and multiple constraint cost subnetworks using a quantile regression architecture. It is used to output the estimated values ​​of cumulative return and cumulative cost at preset quantiles, and to extract risk-sensitive indicators; The soft scheduler module, connected to the multi-player decision-making module and the risk perception evaluation module, is used to generate a satisfaction signal based on the comparison results of the cumulative cost of each constraint and the preset safety threshold. The signal is then converted into a time-varying learning rate coefficient through a smoothing mapping function to modulate the update step size of each policy network.

2. The multi-agent cooperative safety control system based on an adaptive game theory architecture as described in claim 1, characterized in that, The adaptive mixer module specifically includes: The attention weight calculation unit is used to extract global state features and intermediate layer features of each policy network, and calculates the normalized weights through the scaling dot product attention mechanism. The nonlinear weighted fusion unit is used to perform weighted summation and ensure the physical feasibility of the output action through a differentiable projection function; The risk-adaptive gating unit is used to dynamically adjust the fusion weights based on risk-sensitive indicators.

3. A multi-agent cooperative safety control system based on an adaptive game theory architecture as described in claim 2, characterized in that, The risk perception and assessment module includes: The task value subnetwork is used to estimate the cumulative return at a preset quantile. Multiple constraint cost subnetworks are used to estimate the estimated values ​​of each security constraint cost at preset quantiles; The risk indicator extraction unit is used to extract risk-sensitive indicators based on the estimated values ​​of the task value subnetwork and multiple constrained cost subnetworks.

4. A multi-agent cooperative safety control system based on an adaptive game theory architecture as described in claim 3, characterized in that, The soft scheduler module includes: The satisfaction monitoring unit is used to monitor the cumulative cost of each constraint in real time and compare it with a preset safety threshold to generate a continuous satisfaction signal. The smoothing mapping unit is used to map the satisfaction signal into time-varying learning rate coefficients through a parameterized function. The update scheduling unit is used to allocate gradient update dominance between the task agent and each constraint agent according to the learning rate coefficient.

5. A multi-agent cooperative safety control method based on an adaptive game theory architecture, applied to a multi-agent cooperative safety control system based on an adaptive game theory architecture as described in any one of claims 1-4, characterized in that, Includes the following steps: S1: Construct a distributed decision architecture based on Markov game theory, decouple the controlled system into a task agent and multiple constraint agents. The task agent is equipped with a task policy network, and each constraint agent is equipped with an independent constraint policy network. The weight parameters of the task policy network and the constraint policy network are randomly initialized. S2: At each decision moment, the environmental state observation vector is simultaneously input into the task policy network and the constraint policy network to obtain the action suggestion vector; the fusion weight of each action suggestion is dynamically calculated based on the attention mechanism, weighted summation is performed and processed by a differentiable projection function, and the joint action that conforms to the action space boundary is output. After the joint action interacts with the environment, the interaction data is stored in the experience replay pool. S3: Sample batch data from the experience replay pool and input it into the risk perception evaluation module; the risk perception evaluation module contains a parallel task value sub-network and multiple constraint cost sub-networks, all of which adopt a quantile regression architecture and output the estimated values ​​of cumulative return and cumulative cost at preset quantiles respectively; train the risk perception evaluation module based on the quantile loss function and extract risk-sensitive indicators; S4: Generate a satisfaction signal based on the comparison between the cumulative cost of each constraint and the preset safety threshold, and convert it into a time-varying learning rate coefficient through a smoothing mapping function; S5: Based on the risk-sensitive index and the time-varying learning rate coefficient, update the task policy network and each constraint policy network, wherein the update step size of the task policy network and each constraint policy network is modulated by the corresponding learning rate coefficient.

6. The multi-agent cooperative security control method based on an adaptive game theory architecture as described in claim 5, characterized in that, The methods for dynamically calculating fusion weights based on attention mechanisms in S2 include: Extract global state features and features of intermediate layers of each policy network; The correlation score between each policy feature and the global state feature is calculated using a scaled dot product attention mechanism. The correlation scores are normalized into fusion weights using the Softmax function.

7. The multi-agent cooperative security control method based on an adaptive game theory architecture as described in claim 6, characterized in that, S2 also includes a risk-adaptive gating step, specifically: When the risk sensitivity indicators extracted by the risk perception and evaluation module exceed the preset risk threshold, the calculation process of the attention mechanism is dynamically adjusted to dynamically increase the fusion weight corresponding to the safety constraint agent.

8. The multi-agent cooperative security control method based on an adaptive game theory architecture as described in claim 7, characterized in that, In S3, the quantile loss function is the quantile Huber loss function, expressed as: ； in, The target quantile is calculated based on the target network and the Bellman equation. The quantile Huber loss function, It is the preset total number of quantiles. For state-action pairs Find the expected value. The output of the task value subnetwork at the quantile The estimated value.

9. A multi-agent cooperative security control method based on an adaptive game theory architecture as described in claim 8, characterized in that, The satisfaction signal is generated based on the comparison between the cumulative cost of each constraint and the preset safety threshold, specifically including: Real-time monitoring of the cumulative cost of each constrained agent; Calculate the difference between the accumulated cost and the preset safety threshold; By using a buffered clipping function, the gap is mapped to a satisfaction signal within the interval.

10. A multi-agent cooperative security control method based on an adaptive game theory architecture as described in claim 9, characterized in that, The smoothing mapping function is a parameterized Sigmoid-like function that nonlinearly maps the satisfaction signal into time-varying learning rate coefficients.

Images