An AI-based intelligent control system for robot collaboration

By using a nonstationarity quantification and environmental anchoring confidence graph construction module and an adaptive two-layer drift suppression module, the problems of nonstationarity and policy drift in multi-robot collaborative linkage are solved, and the safety and consistency control of robot collaborative actions are achieved.

CN123113536APending Publication Date: 2026-11-17KUNYA (SHANGHAI) INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611351215.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-02
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Non-stationarity and policy drift exist in multi-robot collaborative linkage, leading to unreasonable collaborative actions. Furthermore, when communication is limited, the value function estimation is biased, making it difficult to achieve effective collaborative control.

Method used

A non-stationary quantization and environment-anchored confidence graph construction module is adopted. The policy drift gradient is calculated by Wasserstein distance to generate the environment-anchored confidence graph. Combined with an adaptive two-layer drift suppression module, the policy drift is suppressed by feedforward prediction and feedback constraints. A value consensus verification and action arbitration module is adopted to perform action arbitration when communication is limited.

Benefits of technology

Effectively prevent and suppress strategy drift, ensure the safety and consistency of robot collaborative actions, reduce communication bandwidth requirements, avoid overly aggressive behavior caused by information asymmetry, and achieve global coordination verification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123113536A_ABST
    Figure CN123113536A_ABST
Patent Text Reader

Abstract

This invention discloses an AI-based intelligent control system for robot collaborative linkage, belonging to the field of robot collaborative control technology. By calculating the Wasserstein distance between the probability distributions of actions in adjacent decision cycles and obtaining the rate of change over time, the policy drift gradient of each robot is obtained. Through spatial aggregation, a collaborative non-stationary tensor is constructed, and an environmental anchoring confidence map is generated. Based on the environmental anchoring confidence map, a two-layer collaborative drift suppression system of feedforward prediction and feedback constraint is implemented to construct a policy safety envelope. For actions after feedforward pre-compensation, envelope boundary crossing detection is performed. If boundary crossing still exists, policy bounce correction is triggered, mapping the actions back to the safe region within the envelope, and outputting a set of corrected actions after two-layer drift suppression. Based on the output set of corrected actions, when communication is limited, the local value function of each robot is compressed into a low-dimensional value snapshot summary and transmitted with limited bandwidth.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of robot collaborative control technology, specifically to an AI-based intelligent control system for robot collaborative linkage. Background Technology

[0002] In complex and dynamic environments, multi-agent reinforcement learning (MARL) is often used for decision-making in multi-robot collaboration. However, MARL suffers from non-stationarity: each robot's policy changes continuously during training, causing the environment perceived by other robots to change dynamically, making training difficult to converge or getting stuck in local optima; after deployment, due to environmental distribution shifts, such as new obstacles or new tasks, policies may drift, meaning that robot behavior gradually deviates from the safe zone during training, resulting in unpredictable collaborative actions; furthermore, the centralized training and distributed execution (CTDE) framework suffers from biased value function estimation when communication is limited, leading to unreasonable action selection. These issues highlight the non-stationarity and policy drift defects in multi-agent reinforcement learning for robot collaboration. Summary of the Invention

[0003] The purpose of this invention is to provide an AI-based intelligent control system for robot collaboration and linkage, which solves the technical problems mentioned in the background art.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] An AI-based intelligent control system for robot collaboration includes:

[0006] Nonstationarity quantification and environmental anchoring confidence map construction module: Collect the action probability distribution of each robot's policy network within a continuous decision cycle. By calculating the Wasserstein distance between the action probability distributions of adjacent decision cycles and obtaining the rate of change over time, the policy drift gradient of each robot is obtained. The policy drift gradients of multiple robots are spatially aggregated to construct a cooperative nonstationarity tensor. Based on the cooperative nonstationarity tensor, an environmental anchoring confidence map characterizing the reliability of the current environmental state of each robot is generated.

[0007] Adaptive Two-Layer Drift Suppression Module: Based on the generated environment anchor confidence map, it performs two-layer collaborative drift suppression of feedforward prediction and feedback constraints: First, it uses the historical sequence of policy drift gradients to construct an exponentially weighted trend predictor to estimate the drift direction and magnitude of the next decision cycle. On the original policy output action, it applies a feedforward pre-compensation force in the opposite direction to weaken the drift trend from the source. Then, based on the environment anchor confidence map, it constructs a policy safety envelope that dynamically shrinks or expands with confidence. It performs envelope boundary detection on the action after feedforward pre-compensation. If there is still a boundary violation, it triggers policy rebound correction, maps the action back to the safe area within the envelope, and outputs the corrected action set after two-layer drift suppression.

[0008] Value consensus verification and action arbitration module: Based on the output set of corrected actions, when communication is limited, the local value function of each robot is compressed into a low-dimensional value snapshot summary and transmitted with limited bandwidth. The receiver performs consensus verification with the received value snapshot summary and the local historical value estimate. When the deviation exceeds the preset threshold, the conservative estimation mode is activated to arbitrate the action and output the final coordinated linkage control command.

[0009] Furthermore, during the collaborative operation of multiple robots, each robot is equipped with an independent policy network. In each decision cycle, the policy network outputs an action probability distribution based on the current observation state. The system continuously collects the action probability distribution sequence of each robot in consecutive decision cycles, forming the policy evolution time series of each robot.

[0010] Furthermore, after obtaining the information of each robot in the first... Wasserstein distance over decision cycles Next, the rate of change of this distance between adjacent decision periods is calculated, i.e., the policy drift gradient, which is expressed as follows: ;in, Indicates the first The robot in the first The policy drift gradient over each decision cycle; For the calculated first The robot in the first Wasserstein distance over each decision cycle; For the first The robot in the first Wasserstein distance over each decision cycle; The time interval between adjacent decision cycles; For robot index variables, , This indicates the total number of robots participating in the collaborative operation.

[0011] Furthermore, the drift gradients of each robot are coupled according to their communication topology to construct a cooperative nonstationary tensor, the calculation expression of which is as follows: ;in, Indicates the first The collaborative nonstationary tensor of each decision cycle; Indicates the first The set of communication neighbors of each robot; Indicates the first The robot and the first Interaction weights between robots; For the calculated first The robot in the first The policy drift gradient over each decision cycle; For the first The transpose of the policy drift gradient of a robot.

[0012] Furthermore, the environmental anchoring confidence value that each robot can use independently is calculated, and its calculation expression is as follows: ;in, Indicates the first The robot in the first Environmental anchoring confidence level for each decision cycle; is the confidence decay coefficient, which is a positive real number; Represents the co-nonstationary tensor The Frobenius norm; is a smoothing constant, and is a positive real number; For the calculated first The policy drift gradient of each robot; Indicates the first All decision-making cycles Take the maximum value among the policy drift gradients of each robot; It is a natural exponential function;

[0013] The environmental anchoring confidence values ​​of all robots together constitute the environmental anchoring confidence map.

[0014] Furthermore, after obtaining the predicted drift gradient, in the th... Apply a feedforward pre-compensation force opposite to the predicted drift direction to the original output action of the robot's policy network. This pulls the policy output back before the drift trend has fully developed, thus weakening the cumulative effect of drift at the source.

[0015] Furthermore, after completing the feedforward pre-compensation, the environmental anchoring confidence level of the preceding output is used to provide the first... Each robot constructs a policy safety envelope and determines the dynamic radius of the envelope. The policy safety envelope is a spherical safety region with the historical safe anchor point action as the center and the dynamic radius as the boundary. Any policy output action must fall within this spherical region to be considered safe.

[0016] Furthermore, in the The decision cycle, the first The local value function of a robot is a high-dimensional vector whose dimension is equal to the number of discrete actions in the robot's action space or the number of discretized grids in the continuous action space. To compress this high-dimensional vector into a low-dimensional representation, learnable linear compression is used, which combines mapping with low-dimensional subspace projection.

[0017] Furthermore, when the conservative estimation mode is activated to arbitrate the action and output the final control command, after obtaining the consensus deviation, it is compared with the preset consensus threshold, and different action arbitration strategies are selected based on the comparison result.

[0018] If the consensus deviation does not exceed the threshold, the correction action output by the aforementioned two-layer drift suppression module is directly accepted; if the consensus deviation exceeds the threshold, the conservative estimation mode is activated, and the conservative value is taken between the received remote value estimate and the local conservative value estimate for action arbitration.

[0019] Furthermore, the final control commands for all robots are subjected to global coordination verification, global coordination indices are calculated, and it is determined whether the current set of control commands meets the system-level coordination requirements.

[0020] Compared to existing solutions, the beneficial effects achieved by this invention are:

[0021] This invention avoids the problem of metric divergence when the distributed support sets do not overlap by employing Wasserstein distance; it uses time difference operations to extract drift gradients, enabling early detection of the acceleration trend of policy drift; it uses outer product space aggregation to retain the correlation information of drift direction and identify collective drift patterns; it uses an exponential mapping with a two-factor product structure to generate confidence, taking into account both the dual impact of system-level non-stationarity and individual-level drift severity; the feedforward layer uses an exponentially weighted trend predictor to estimate the drift direction and amplitude in advance, transforming traditional ex-post correction into ex-ante prevention, eliminating the lag defect of fixed threshold schemes where correction is triggered only when drift accumulates to a dangerous level; the feedback layer achieves adaptive matching between the strength of safety constraints and the degree of environmental credibility by dynamically shrinking or expanding the safety envelope surface by anchoring the confidence with the environment, maintaining sufficient degrees of freedom of action when the environment is credible, and tightening the safety boundary when the environment is untrustworthy. This approach avoids the inherent contradiction between security and flexibility inherent in fixed security boundaries. Through learnable compression mapping, high-dimensional value functions are compressed into low-dimensional value snapshot summaries, significantly reducing communication bandwidth requirements while preserving core decision-making information. This resolves the contradiction between limited communication bandwidth and the integrity of value information in multi-robot collaborative operations. A consensus verification mechanism detects value perception discrepancies among multiple robots with extremely low communication overhead, promptly identifying potential collaborative decision-making conflicts even under incomplete information conditions. A maximum-minimum conservative arbitration strategy uses conservative lower bounds of remote and local estimates for decision-making when value perception discrepancies arise, ensuring safe and controllable actions even in the most pessimistic scenario, avoiding overly aggressive behavior due to information asymmetry. Global coordination verification serves as the final safety checkpoint, ensuring that the independent arbitration results of each robot satisfy collaborative consistency at the global level, preventing collaborative conflicts caused by heterogeneous arbitration. Attached Figure Description

[0022] The invention will now be further described with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart illustrating the operation of an AI-based intelligent control system for robot collaboration and linkage according to the present invention.

[0024] Figure 2 This is a block diagram of an AI-based intelligent control system for robot collaboration and linkage according to the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] like Figures 1 to 2 As shown, this invention is an AI-based intelligent control system for robot collaboration, comprising:

[0027] Nonstationarity quantification and environmental anchoring confidence map construction module: Collect the action probability distribution of each robot's policy network within a continuous decision cycle. By calculating the Wasserstein distance between the action probability distributions of adjacent decision cycles and obtaining the rate of change over time, the policy drift gradient of each robot is obtained. The policy drift gradients of multiple robots are spatially aggregated to construct a cooperative nonstationarity tensor. Based on the cooperative nonstationarity tensor, an environmental anchoring confidence map characterizing the reliability of the current environmental state of each robot is generated.

[0028] When collecting the action probability distribution of each robot's policy network within a continuous decision-making cycle, in the process of multi-robot collaborative operation, each robot is equipped with an independent policy network. This policy network outputs an action probability distribution based on the current observation state in each decision-making cycle; specifically, at time... , No. The robot's policy network receives the current environmental observation state. Output the robot's motion space Action probability distribution , Indicates the first Within the decision-making cycle, the first The robot selects an action. The probability value, ; For robot index variables, , This indicates the total number of robots participating in the collaborative operation.

[0029] Among them, the The first robot One decision cycle, denoted as This refers to the first At the end of the action execution of the decision cycle, until the [number]th [cycle]... The time span between the completion times of actions in each decision cycle; each decision cycle Internally, it is divided into three sequentially executed sub-stages, specifically:

[0030] The first stage is the status acquisition and preprocessing stage, the duration of which is denoted as [duration]. During this phase, the robot completes the reading of all sensor data, timestamp synchronization, coordinate transformation, and feature extraction operations, ultimately generating the current environmental observation status. The starting time of this stage is the [number]th [stage]. The start of a decision cycle ;

[0031] The second stage is the strategic reasoning stage, and its duration is denoted as [duration]. During this phase, the airborne computing unit will generate... Input Policy Network Perform a complete forward propagation calculation and output the action probability distribution. and according to Sampling or determining the actions expected to be performed this week This phase follows immediately after the first phase, and its starting time is... ;

[0032] The third stage is the action execution and feedback stage, the duration of which is denoted as [duration missing]. During this phase, the underlying controller will take action. This signal is converted into a motor drive signal and continuously applied to the actuator. Simultaneously, feedback data from the actuator itself is collected during execution for state estimation in the next cycle. This stage immediately follows the second stage, and its end time is the [missing information - likely a specific timeframe]. The end time of each decision cycle ;

[0033] The sum of the durations of the above three sub-stages constitutes the actual duration of the decision-making cycle, i.e. ;

[0034] In addition, the current environmental observation status It is not a single scalar, but a composite high-dimensional vector that integrates its own ontological information, local environmental perception information, and multi-agent collaborative interaction information; specifically, It is composed of the following three types of sub-state vectors: ;in, For the first The robot's own body state subvector, specifically including: the robot's three-dimensional position coordinates in the global coordinate system. Three-dimensional attitude angles (roll angle) Pitch angle Yaw angle ), three-axis velocity Triaxial angular velocity and the remaining battery percentage The above data was collected in real time by the inertial measurement unit (IMU), global navigation satellite system (GNSS) receiver, wheel speed meter or odometer, and battery management system (BMS) carried by the robot. For the first The local environment perception state sub-vector of a robot specifically includes: a local occupancy grid map or point cloud feature vector with the robot itself as the origin, used to represent the spatial distribution of surrounding obstacles; and the task target point's relative position vector relative to the robot. ; and environmental semantic label encoding, such as one-hot encoded vectors of discrete features like terrain type and lighting conditions; the above data is collected by environmental perception sensors such as lidar, depth camera, and RGB camera carried by the robot, and obtained after preprocessing and feature extraction by the onboard computing unit. These are all existing conventional technical solutions, and the specific implementation steps will not be elaborated here. For the first The cooperative interaction state sub-vector of each robot specifically includes: the set of communicating neighbors. Each neighbor robot Compared to the first The relative positions of the robots Relative velocity The data includes the task status identifier broadcast by neighboring robots; if the number of communicating neighbors changes dynamically, a fixed-length padding vector or attention pooling mechanism is used to compress the variable-length neighbor information into a fixed-length sub-vector; the above data is received and parsed through wireless communication modules between robots, such as Wi-Fi, 5G or ad hoc radios.

[0035] The splicing order and dimension of the above three types of sub-state vectors remain fixed throughout the entire operation cycle of the multi-robot system, ensuring that the input dimension of the policy network is constant, thereby ensuring the consistency of the domain of action probability distribution in the subsequent Wasserstein distance calculation.

[0036] To obtain the current environmental observation status After that, the Probability distribution of robot actions It is obtained through forward inference computation of its airborne policy network; the specific generation process is as follows:

[0037] The policy network employs a parameterized deep neural network structure, denoted as... ,in, This is the set of learnable weight parameters for the policy network; the network is based on the current environmental observation state. As input, after multiple nonlinear transformations, the output parameters are the probability distribution of the action; based on the action space... Different types, The generation methods fall into the following two categories:

[0038] When the action space When dealing with a discrete action space, such as high-level behavior selection for a robot, including a finite number of discrete actions like "forward," "stop," "turn left," and "turn right," a policy network is needed. The output layer is a vector with dimension equal to the number of discrete actions. After being processed by the Softmax normalization function, the selection probability of each discrete action is directly obtained. ,in, This represents the forward propagation function of the policy network excluding the output layer. This represents the action-related vector in the output vector of the Softmax function. The amount, satisfying and ;

[0039] When the action space When dealing with continuous motion spaces, such as the low-level motion control of a robot, including continuous values ​​of linear and angular velocities, a policy network is needed. The Gaussian strategy parameterization is used, and its output layer contains two sub-output heads: the mean head and the mean head. and logarithmic standard deviation head At this point, the probability distribution of actions , as a As the mean, with A multidimensional Gaussian distribution with standard deviation, i.e. For any action in the continuous action space Its probability density value is given by the Gaussian probability density function; in the actual calculation of the Wasserstein distance, for a continuous Gaussian distribution, analytical formulas or Monte Carlo sampling can be used for approximate calculation.

[0040] It should be noted that the weight parameters of the policy network... In the online deployment phase of multi-robot collaborative operations, updates are typically fixed, either frozen or only fine-tuned online at extremely low frequencies; therefore, the probability distribution of actions between adjacent decision cycles is... and The changes mainly stem from the environmental observation status. The changes in parameters, and the possible online adaptive updates, lead to small parameter drifts rather than changes in the policy network structure. This distribution change caused by both state-driven and parameter drift is the source of the policy drift signal that Wasserstein distance aims to capture.

[0041] The system continuously collects the action probability distribution sequence of each robot within a continuous decision-making cycle, i.e. This forms the time series of strategy evolution for each robot;

[0042] Obtain the action probability distribution of two adjacent decision cycles and Next, the Wasserstein distance between the two is calculated. The Wasserstein distance, also known as the bulldozer distance, measures the minimum cost required to transform one probability distribution into another. It captures changes in the global shape of the action probability distribution. Compared to point-by-point measures such as KL divergence, the Wasserstein distance maintains good numerical stability and physical interpretability even when the distribution support sets do not overlap. It more accurately reflects the overall offset of the policy network output in the action space. Its calculation expression is: ;in, Indicates the first The robot in the first The Wasserstein distance between the action probability distribution of each decision cycle and the previous decision cycle is used to quantify the overall offset of the robot's policy output in adjacent decision cycles. Indicates the first The robot in the first The probability distribution of actions in each decision cycle; Indicates the first The robot in the first The probability distribution of actions in each decision cycle; This represents the robot's action space, which is the set of all possible actions. The Cartesian product of the action space is used to define the search domain for the joint transport. Indicated by and It is the set of all joint transmission schemes with a marginal distribution, where each joint transmission scheme is a coupled distribution. , representing a kind of The probability mass above, moved to The above method; In terms of action space, two actions and The distance between them is usually measured by Euclidean distance in continuous action space, and Hamming distance or other predefined metric can be used in discrete action space. Let be the distance order, which is a positive integer, typically taken as . or , The value of determines the sensitivity to transmission cost. The larger the value, the heavier the penalty for long-distance transmission; Indicates joint transmission scheme In action pairs On the infinitesimal element; This means finding the minimum among all feasible joint transport schemes, that is, finding the optimal transport path that minimizes the total transport cost.

[0043] It is important to note that the Wasserstein distance transforms the distribution changes of the policy network within adjacent decision periods into a scalar value, providing a basic metric for subsequent quantification of policy drift rate. Compared to directly comparing policy network parameters or the Euclidean distance of output actions, the Wasserstein distance characterizes the global features of policy changes at the distribution level. This effectively avoids measurement distortion caused by different parameter update directions within the policy network but consistent output distribution change directions, thus improving the accuracy and robustness of drift perception.

[0044] When calculating the time rate of change of the Wasserstein distance to obtain the policy drift gradient of each robot, the time rate of change of the Wasserstein distance is obtained at the time when the robot is in the first phase. Wasserstein distance over decision cycles Next, the rate of change of this distance between adjacent decision periods is calculated, i.e., the policy drift gradient. The policy drift gradient reflects the acceleration or deceleration trend of policy deviation: if the drift gradient is positive and continuously increasing, it indicates that the policy is accelerating away from the stable region during training; if the drift gradient is negative, it indicates that the policy is regressing towards the stable region. Its calculation expression is: ;in, Indicates the first The robot in the first The policy drift gradient for each decision cycle, in units of action space distance divided by seconds, is used to characterize the rate of change of the robot's policy offset over time. For the calculated first The robot in the first Wasserstein distance over each decision cycle; For the first The robot in the first The Wasserstein distance for each decision cycle is the distribution offset of the previous cycle calculated using the same method in the preceding steps; when When, it indicates the first The robot's strategy is accelerating the drift; when When, it indicates that the drift is slowing down; when When the time is right, it indicates that the strategy is in a steady state or a constant-speed drift state; The time interval between adjacent decision cycles is expressed in seconds.

[0045] It is important to note that converting static distribution offsets into dynamic drift rate information, relying solely on the absolute value of the Wasserstein distance, cannot distinguish between two distinct risk states: slow drift with a large amplitude and rapid drift with a small amplitude. However, the policy drift gradient, by introducing time-differential operations, can promptly capture the accelerating trend of policy offsets and issue early warning signals before the drift accumulates to a dangerous level, providing the trend information needed for feedforward compensation by the subsequent drift suppression module.

[0046] When spatially aggregating the policy drift gradients of multiple robots to construct a cooperative nonstationary tensor, the policy drift gradient of each robot is obtained. Next, it is necessary to further consider the mutual influence relationships among multiple robots. In the non-stationary scenario of multi-agent reinforcement learning, the policy change of one robot will affect the perception and decision-making of other robots through environmental coupling, such as shared space and cooperative task dependence. Therefore, the drift gradient of a single robot cannot be viewed in isolation. This step uses spatial aggregation operation to couple the drift gradients of each robot according to their communication topology to construct a cooperative non-stationary tensor, the calculation expression of which is: ;in, Indicates the first The cooperative nonstationarity tensor for each decision cycle is a matrix-form aggregate quantity used to comprehensively characterize the nonstationarity level and spatial distribution structure of the entire multi-robot cooperative system at that moment. Indicates the first The communication neighbor set of the i-th robot, i.e., with the i-th robot. In a communication topology diagram, a robot is the set of other robots that are directly connected to it, and this set is determined by the communication network structure of the multi-robot system. Indicates the first The robot and the first The interaction weights between the robots are calculated by attenuating the weights based on the physical distance between the two robots. Specifically, ,in For the first The robot and the first The Euclidean distance between the robots The distance decay feature length is used to control the rate at which the interaction effect decays with distance; For the calculated first The robot in the first The policy drift gradient over each decision cycle; For the first The transpose of the policy drift gradient of each robot is multiplied to form an outer product, which is used to capture the correlation and coupling strength between the drift directions of the two robots; superscript This represents the vector transpose operation.

[0047] The expression uses the outer product form. The logic is as follows: when the drift gradient directions of two adjacent robots are consistent, the eigenvalues ​​of the outer product matrix are large, indicating that the cooperative system has a collective drift risk in that direction; when the drift gradient directions of two adjacent robots are orthogonal or opposite, the corresponding elements in the outer product matrix tend to zero or are negative, indicating that the drifts of the two robots cancel each other out or are independent, and the overall nonstationarity of the cooperative system is low; by weighted summation of the outer products of all communicating neighbor pairs, the cooperative nonstationarity tensor... This comprehensively reflects the non-stationary distribution structure of multi-robot systems in the spatial dimension.

[0048] The purpose of this step is to integrate the independent drift information of each robot into a system-level nonstationarity representation. In multi-agent reinforcement learning, the core source of nonstationarity is the synchronous change of the strategies of each agent, which leads to continuous changes in the dynamics of the environment. The drift gradient of a single robot only reflects local information, while the cooperative nonstationarity tensor, through spatial aggregation, elevates local information into global information. This enables the identification of the most dangerous cooperative failure mode, where multiple robots drift simultaneously, providing a system-level basis for subsequent confidence assessment.

[0049] When generating the environmental anchoring confidence map based on the cooperative nonstationary tensor, the cooperative nonstationary tensor is obtained... This is then further transformed into an environment anchoring confidence value that each robot can use independently. The environment anchoring confidence reflects the reliability of the current environment state for that robot: when cooperative nonstationarity is high and the robot's own drift gradient is large, the confidence decreases, meaning the current environment state may have deviated from the training distribution, and the reliability of the policy output decreases; conversely, when nonstationarity is low and the drift gradient is small, the confidence is close to 1, indicating that the environment state is consistent with the training distribution and the policy output is reliable. Its calculation expression is: ;in, Indicates the first The robot in the first The environmental anchoring confidence level for each decision cycle, with a value range of [value range missing]. ,when When the value approaches 1, it indicates that the environmental condition is highly reliable. When the value approaches 0, it indicates that the environment state deviates significantly from the training distribution and the policy output is unreliable; is the confidence decay coefficient, a positive real number that controls the sensitivity of confidence to decrease as nonstationarity increases. The larger the confidence level, the more sensitive it is to non-stationary changes. Represents the co-nonstationary tensor The Frobenius norm, which is the square root of the sum of squares of all elements in a tensor, is used to compress nonstationarity information in matrix form into a single scalar to measure the strength of the overall nonstationarity of the system. is a smoothing constant, a positive real number, used to avoid when As the expression approaches zero, its numerical value becomes unstable, while simultaneously acting as a normalizer, making... The value of is always in Within the range; For the calculated first The policy drift gradient of each robot; Indicates the first All decision-making cycles The maximum value among the policy drift gradients of the i-th robot is taken and used to adjust the i-th robot's gradient. The drift gradient of each robot is normalized so that... The value of is in Within the interval, representing the first The degree of drift of each robot relative to the maximum degree of drift within the system; This is the natural exponential function, used to map negative values ​​within the exponent to... The confidence level of the interval.

[0050] The logical structure of this expression is as follows: the exponent is composed of the product of two factors, the first factor being... The second factor reflects the overall nonstationarity level of the system. Reflecting the first The relative severity of individual robot drift within the system; the two factors are multiplied together, and then multiplied by the attenuation coefficient. The negative exponent is used so that when the system is highly non-stationary and the robot drifts severely, the confidence level drops sharply, while when the system is stable or the robot drifts slightly, the confidence level remains at a high level.

[0051] Environmental anchoring confidence values ​​for all robots Together, they form an environmental anchoring confidence map. This confidence map uses the spatial position of each robot as nodes and the communication topology as edges to form a confidence distribution map that is dynamically updated over time. This provides direct quantitative input for the dynamic contraction and expansion of the safety envelope in the subsequent adaptive two-layer drift suppression module.

[0052] The purpose of this step is to reduce the information dimensionality from global diagnosis to individual assessment by transforming system-level nonstationarity information into a confidence scalar that each robot can interpret independently. The environment-anchored confidence map combines global relevance and individual specificity: global relevance is reflected in the use of system-level cooperative nonstationarity tensor norms in the confidence calculation; individual specificity is reflected in the fact that each robot's confidence is also modulated by the ratio of its own drift gradient to the maximum drift of the system. This design avoids a one-size-fits-all global uniform threshold judgment, allowing robots with less drift to still maintain a large degree of policy freedom, while robots with more drift are subject to stricter constraints, achieving differentiated and refined reliability assessment.

[0053] In this embodiment, through the sequential execution of the above steps, the policy drift gradient perception and non-stationarity quantification module achieves a progressive processing from single-robot distribution offset measurement to multi-robot system-level non-stationarity quantification, and then to individualized environmental reliability assessment. By using Wasserstein distance instead of KL divergence or JS divergence for distribution offset measurement, the problem of metric value divergence when the distribution support sets do not overlap is avoided. Temporal difference operations are used to extract the drift gradient, rather than just using the absolute value of the offset, enabling early detection of the acceleration trend of policy drift. Outer product space aggregation is employed, while... Instead of simple summation or averaging, this approach preserves the correlation information of drift direction and identifies collective drift patterns. The exponential mapping using a two-factor product structure generates confidence scores, balancing the dual impacts of system-level non-stationarity and individual-level drift severity. This design enables the module to accurately, timely, and differentiatedly perceive policy drift and non-stationarity risks in complex dynamic environments with multi-robot collaboration, providing a reliable quantitative basis for subsequent drift suppression and action arbitration. This effectively solves the technical problems of difficulty in quantifying non-stationarity and early warning of policy drift in multi-agent reinforcement learning.

[0054] Adaptive Two-Layer Drift Suppression Module: Based on the generated environment anchor confidence map, it performs two-layer collaborative drift suppression of feedforward prediction and feedback constraints: First, it uses the historical sequence of policy drift gradients to construct an exponentially weighted trend predictor to estimate the drift direction and magnitude of the next decision cycle. On the original policy output action, it applies a feedforward pre-compensation force in the opposite direction to weaken the drift trend from the source. Then, based on the environment anchor confidence map, it constructs a policy safety envelope that dynamically shrinks or expands with confidence. It performs envelope boundary detection on the action after feedforward pre-compensation. If there is still a boundary violation, it triggers policy rebound correction, maps the action back to the safe area within the envelope, and outputs the corrected action set after two-layer drift suppression.

[0055] Using the first A robot passed by Historical sequence of policy drift gradients over a decision cycle An exponentially weighted trend predictor is constructed to estimate the drift direction and magnitude of the robot in the next decision cycle. The next decision cycle is the [missing information]. The calculation expression for each decision cycle is: ; ;in, Indicates the first The robot in the first The predicted drift gradient for each decision cycle contains information in two dimensions: drift direction and drift magnitude. The direction indicates the direction of the action space that the policy will deviate from, and the magnitude indicates the severity of the deviation. The historical window length is a positive integer representing the number of historical drift gradient samples used for prediction. The larger the value, the smoother the prediction, but the slower the response to mutations. The smaller the value of , the more sensitive the response to mutations, but the greater the prediction noise; For the first The normalized index weight corresponding to each historical sample represents the proportion of that historical sample in the prediction. For the index number of the historical samples, Corresponding to the most recent decision-making cycle, i.e. the [number]th One cycle, Corresponding to the earliest decision-making cycle, namely the first One cycle; Indicates the first The robot in the first The policy drift gradient for each decision cycle, i.e., the historical drift gradient value calculated by the aforementioned steps; The forgetting factor has a value range of 1. Control the rate at which historical information decays. The closer it is to 1, the greater the weight of recent samples and the faster the weight of distant samples decays, making the predictor more sensitive to recent changes. The closer it is to 0, the more even the weights of each historical sample become, and the more the predictor tends to make a smooth estimate. A dummy index in normalized summation, used to calculate the normalized denominator of all weights, ensuring that all... The sum equals 1; It uses an exponentially decaying kernel, which gives higher weights to more recent drift gradient samples and decreases the weights of more distant samples exponentially.

[0056] The design logic of this expression is as follows: policy drift usually has temporal continuity and trend inertia. That is, if the policy continues to drift in a certain direction in the near future, it is highly likely to continue to drift in that direction in the next decision cycle, but the drift rate may change due to environmental changes. The exponential weighting structure gives higher weight to recent samples, which can track the latest changes in the drift trend. At the same time, the weighted average of multiple historical samples suppresses the interference of single-cycle noise, achieving a balance between trend tracking and noise suppression. The normalization process ensures that the sum of the weights is 1, so that the dimension of the predicted drift gradient is consistent with the original drift gradient, which is convenient for subsequent direct use in action compensation calculation.

[0057] The purpose of this step is to transform passive post-event detection into proactive pre-event prediction. Traditional drift suppression schemes only make corrections after drift has occurred and caused out-of-bounds consequences, resulting in a response lag problem. This step, however, estimates the drift direction and magnitude of the next cycle in advance through trend prediction, providing directional guidance and magnitude reference for subsequent feedforward pre-compensation. This allows the system to apply a reverse correction force before drift actually occurs, eliminating lag in the time dimension and achieving proactive drift prevention in a preventative manner.

[0058] In obtaining the predicted drift gradient Then, this step is in the... A feedforward pre-compensation force, opposite to the predicted drift direction, is applied to the original output action of the robot's policy network. This pulls the policy output back before the drift trend fully develops, weakening the cumulative effect of drift at its source. The calculation expression is as follows: ;in, Indicates the first The robot in the first The action after feedforward pre-compensation in each decision cycle will serve as the input to the subsequent feedback envelope constraint layer. Indicates the first The policy network of the robots in the th... Each decision cycle contains the original output action without any modification, which is the action value directly output by the policy network based on the current observation state; The pre-compensation strength coefficient is a positive real number that controls the magnitude of the feedforward compensation force. The larger the value, the stronger the compensation and the more aggressive the suppression of drift, but an excessively large value... This could lead to overcorrection, causing the action to deviate from the reasonable range. The value of needs to be calibrated according to the typical range of drift rate in the specific application scenario; To estimate the obtained first The predicted drift gradient for the next decision cycle of the robot, whose direction indicates the direction of the drift trend and whose magnitude indicates the strength of the drift trend; Indicates the first The strategy function of each robot For action variables The gradient vector, in the current observation state The gradient vector, calculated below, indicates the direction in which the policy probability density increases the fastest in the action space, and is used to determine the direction of application of the compensation force in the action space. Indicates the first The robot in the first The current environmental observation status of each decision cycle includes observation data such as the robot's own pose, speed, surrounding obstacle information, and the status of neighboring robots; The magnitude of the policy gradient vector, i.e., the Euclidean norm, is used to normalize the gradient direction. Let be the numerical stability constant, a small positive quantity, for example, take . This is added to the denominator to prevent division by zero when the policy gradient approaches the zero vector, thus ensuring the numerical stability of the calculation; fraction The overall representation is a normalized vector of the policy gradient direction, ensuring that the direction information of the compensation force is not affected by the magnitude of the gradient, and only the direction attribute is retained.

[0059] The physical logic of this expression is as follows: the feedforward pre-compensation force is composed of the product of three factors, including the pre-compensation strength coefficient. Controlling compensation intensity and predicting drift gradient It provides a reference for the direction and magnitude of compensation, and the unit vector of the policy gradient determines the specific direction of the compensation force in the action space; the negative sign indicates that the direction of the compensation force is opposite to the direction of the predicted drift, that is, if the predicted policy will drift in a certain direction, the compensation force applies a push in the opposite direction; the magnitude of the compensation force is proportional to the magnitude of the predicted drift. The stronger the drift trend, the greater the compensation. When the drift trend is weak, the compensation naturally decreases, avoiding excessive intervention when there is no drift.

[0060] It needs to be explained that: as the first layer in the two-layer drift suppression architecture, namely the feedforward layer, it implements proactive preventive soft constraints on policy drift; unlike the passive waiting for drift to occur before correction, the feedforward pre-compensation applies reverse correction before the drift accumulates to a dangerous level, eliminating the drift in its infancy; at the same time, since the compensation force is proportional to the predicted drift amplitude, when the predicted drift is zero or extremely small, the compensation force also approaches zero, which will not interfere with the normal policy output, ensuring that the robot's motion accuracy is not affected in a stable operating state.

[0061] After completing the feedforward pre-compensation, this step utilizes the environmental anchoring confidence level of the preceding output. , for the first Each robot constructs a policy safety envelope and determines its dynamic radius. The policy safety envelope is a spherical safety region centered on historical safe anchor actions and bounded by the dynamic radius. Any policy output action must fall within this spherical region to be considered safe. The expression for calculating the dynamic radius is: ;in, Indicates the first The robot in the first The dynamic radius of the policy safety envelope for each decision cycle determines the range of motion offsets currently allowed for the robot. Let be the basic envelope radius, a positive real number, representing the maximum allowable range of motion deviation for the robot under fully trustworthy conditions, where the confidence level is 1. The value of is determined based on engineering parameters such as the robot's physical dimensions, maximum speed, and task accuracy requirements. The first one is calculated in the preceding steps. The robot in the first The environmental anchoring confidence level for each decision cycle, with a value range of [value range missing]. This reflects the reliability of the robot under the current environmental conditions; The contraction sensitivity index is a positive real number that controls the nonlinearity of the envelope radius as a function of confidence level. At this time, the envelope contracts slowly in the early stage of confidence decline and contracts sharply when the confidence is extremely low, making it suitable for scenarios with extremely high safety margin requirements; At this time, the envelope surface shrinks rapidly in the early stage of confidence decline, which is suitable for scenarios with zero tolerance for drift; The time envelope radius is linearly related to the confidence level.

[0062] The logic of this expression is: when the environment anchors the confidence level When the value is close to 1, it indicates that the environmental state is reliable and the non-stationarity is low. Approaching 1, dynamic radius Approximately the basic envelope radius The robot has a large degree of freedom of movement, which will not excessively restrict its ability to perform normal tasks; when the environment is anchored with confidence... When the condition decreases, it indicates that the environmental state is unreliable and highly non-stationary. Significantly reduced, dynamic radius The robot's movements are confined to a smaller safe area by corresponding contraction, reducing the risk of unpredictable actions due to environmental distribution shifts. This dynamic contraction and expansion mechanism allows the strictness of safety constraints to adaptively match the credibility of the current environment, avoiding the inherent contradiction between safety and flexibility in fixed threshold schemes.

[0063] It needs to be explained that: as the foundation of the second layer (feedback layer) in the two-layer drift suppression architecture, a safety boundary is established that adapts to the credibility of the environment; this safety envelope is not statically fixed, but is linked in real time with the environmental anchor confidence output by the aforementioned modules, forming an adaptive mechanism in which the more untrustworthy the environment, the tighter the safety boundary, providing a dynamic reference benchmark for subsequent out-of-bounds detection and bounce correction.

[0064] The envelope surface of the feedforward pre-compensated action is checked for out-of-bounds error. If an out-of-bounds error is detected, the policy bounce correction is triggered after obtaining the feedforward pre-compensated action. Dynamic envelope radius Next, this step performs envelope surface boundary detection on the feedforward pre-compensated action; specifically, it calculates the distance between the feedforward pre-compensated action and the historical safe anchor point action, and compares this distance with the dynamic envelope radius.

[0065] If the distance does not exceed the dynamic envelope radius, it indicates that the action is still within the safe envelope and no further correction is needed; if the distance exceeds the dynamic envelope radius, it indicates that even after feedforward pre-compensation, the action has still exceeded the safe boundary. In this case, policy bounce correction is triggered, pulling the action back to the envelope boundary along the out-of-bounds direction; the criteria for out-of-bounds detection are:

[0066] like If the value exceeds the limit, it is considered out of bounds and a bounce correction is triggered; otherwise, it is considered not out of bounds and the value is directly taken. ;in, For the calculated first The robot's actions after feedforward pre-compensation; For the first The historical safety anchor point actions of the robot are the sliding average of the strategy output actions of the robot during the most recent stable operation period, representing the typical action position of the robot in a safe state; the most recent stable operation period is the period during which the environmental anchor confidence level is continuously higher than the preset safety threshold. The calculated dynamic envelope radius; This represents the Euclidean distance in the action space.

[0067] When an out-of-bounds error is detected, a policy bounce correction is executed, the calculation expression of which is: ;in, Indicates the first The robot in the first Each decision cycle, after rebound correction, results in the final safe action, which is located on the boundary of the safety envelope and in the same direction as the out-of-bounds direction; The historical safety anchor point serves as the reference origin for rebound correction. The dynamic envelope radius determines the maximum allowable distance from the anchor point after the rebound. The offset vector of the action relative to the anchor point after feedforward pre-compensation indicates the direction of the out-of-bounds movement; The magnitude of the offset vector is the actual distance from the anchor point after feedforward pre-compensation; fraction For unit vectors in the out-of-bounds direction, retain the out-of-bounds direction information but eliminate the amplitude information; Multiplying by the unit vector yields the position offset along the cross-boundary direction on the envelope boundary.

[0068] The geometric meaning of this rebound correction is: project the out-of-bounds action back onto the boundary of the envelope surface along the direction in which it deviates from the anchor point, keeping the offset direction unchanged, but truncating the offset amplitude to the dynamic envelope radius. This correction strategy of maintaining the direction and truncating the amplitude eliminates the risk of going out of bounds while preserving the direction information of the action to the greatest extent, avoiding abrupt changes in the robot's motion direction due to completely discarding the out-of-bounds action, and ensuring the continuity and smoothness of the motion.

[0069] The role of this step is as follows: as the core execution link of the feedback layer in the two-layer drift suppression architecture, it performs a fallback hard constraint correction for the out-of-bounds actions that still remain after feedforward pre-compensation; the feedforward layer is a soft constraint, which applies a reverse compensation force through trend prediction, but it cannot guarantee the complete elimination of drift due to limitations in prediction accuracy; the feedback layer is a hard constraint, which ensures that the actions never exceed the safety boundary through geometric projection. The two layers are connected to form a progressive safety mechanism that prioritizes the reduction of soft constraints and provides a fallback guarantee of hard constraints, which ensures both the accuracy of actions under normal conditions and the safety bottom line under extreme conditions.

[0070] When performing cooperative consistency verification on the corrected actions of each robot, the corrected actions of each robot after double-layer drift suppression are obtained. Subsequently, due to the collaborative operation of multiple robots, the actions of each robot must satisfy collaborative constraints, such as formation maintenance, collision avoidance, and consistent task allocation. The safety of individual robot actions cannot guarantee system-level collaborative safety. This step performs a collaborative consistency check on the corrected actions of all robots to ensure that the corrected action set satisfies the collaborative constraints among the robots. The check expression is: ;in, Indicates the first Each decision cycle contains a set of corrected actions after double-layer drift suppression, including all robot actions that have passed the cooperative consistency check. This set will serve as the input for subsequent modules. For the obtained first The robot's corrected motion after undergoing double-layer drift suppression; For the first The robot's corrected motion after undergoing double-layer drift suppression; For the first The communication neighbor set of the i-th robot, consistent with the definition in the preceding steps, represents the set of communication neighbors of the i-th robot. A set of robots that are directly connected in a communication topology; Indicates the first The robot and the first The Euclidean distance between the corrective actions of two robots is used to measure the coordination deviation between their actions. The coordination consistency tolerance threshold is a positive real number representing the maximum allowed cumulative coordination deviation. When the sum of the action deviations of a robot to all its communicating neighbors exceeds this threshold, it indicates that the robot's actions have deviated too much from the coordinated formation and need to be removed from the correction action set, triggering re-coordination.

[0071] The validation logic for this expression is as follows: For each robot, calculate the sum of the distances between its corrected action and the corrected actions of all its communicating neighbors. If this sum of distances does not exceed the collaborative consistency tolerance threshold... Then the robot's corrective action is included in the corrective action set. If the threshold is exceeded, the action does not meet the collaborative constraints and requires further adjustment, or triggers local replanning.

[0072] The purpose of this step is to elevate individual-level safety constraints to system-level collaborative safety constraints. In multi-robot collaborative operations, even if each robot's actions are within its own safety envelope, if the actions between robots are not coordinated, such as two adjacent robots simultaneously shifting significantly in opposite directions, collisions or task failures may still occur. Collaborative consistency verification ensures that the corrected action set not only meets individual safety requirements but also system-level collaborative requirements by checking the cumulative amount of action deviations between robots, thus providing subsequent modules with a candidate set of actions that have undergone dual safety assurance.

[0073] In this embodiment, through the sequential execution of the above steps, a three-level progressive drift suppression is achieved: feedforward prediction and pre-compensation, feedback envelope constraint, and collaborative consistency verification. The feedforward layer, through an exponentially weighted trend predictor, estimates the drift direction and amplitude in advance, transforming traditional ex-post correction into ex-ante prevention, eliminating the lag defect in fixed threshold schemes where correction is only triggered when drift accumulates to a dangerous level. The feedback layer, by anchoring confidence levels according to the environment and dynamically shrinking or expanding the safety envelope, achieves adaptive matching between the strength of safety constraints and the degree of environmental confidence. When the environment is reliable, it maintains sufficient degrees of freedom of action; when the environment is unreliable, it tightens the safety boundary, avoiding the inherent contradiction between safety and flexibility in fixed safety boundaries. The feedforward and feedback layers work together in series to form... The design employs a progressive mechanism that prioritizes soft constraints for reduction and uses hard constraints as a fallback. The feedforward layer reduces most drift trends, while the feedback layer handles residual out-of-bounds movements. This two-layer collaboration ensures drift suppression without being overly aggressive (avoiding disruption to normal task execution) or leaving any safety blind spots (ensuring hard constraints still provide protection in extreme situations). Cooperative consistency verification extends individual safety constraints to system-level cooperative constraints, ensuring that multiple robots remain safe while avoiding cooperative conflicts. This design effectively solves the technical problem in multi-agent reinforcement learning where policy drift, after deployment, gradually deviates from the safe zone due to environmental distribution shifts, leading to unpredictable cooperative actions. It achieves safe and controllable operation of multi-robot collaborative linkage without over-reliance on communication bandwidth.

[0074] Value consensus verification and action arbitration module: Based on the output set of corrected actions, when communication is limited, the local value function of each robot is compressed into a low-dimensional value snapshot summary and transmitted with limited bandwidth. The receiver performs consensus verification with the received value snapshot summary and the local historical value estimate. When the deviation exceeds the preset threshold, the conservative estimation mode is activated to arbitrate the action and output the final coordinated linkage control command.

[0075] When communication is limited, the local value functions of each robot are compressed into low-dimensional value snapshot summaries to adapt to limited bandwidth transmission. In the process of multi-robot collaborative linkage, each robot independently maintains a local value function based on its own observations to evaluate the long-term expected return of each candidate action in the current state. However, in actual deployment environments, the communication bandwidth between robots is often limited by factors such as physical channel capacity, electromagnetic interference, and communication distance attenuation, making it impossible to transmit the high-dimensional complete value function in its entirety between robots. Therefore, this step performs lossy compression on the local value function of each robot and maps it to a low-dimensional value snapshot summary, so that a compressed representation that sufficiently represents the core information of the value function can still be transmitted under limited bandwidth conditions.

[0076] Specifically, in the first The decision cycle, the first Local value function of a robot Let be a high-dimensional vector, whose dimension is equal to the number of discrete actions in the robot's action space, or the number of discretized grids in the continuous action space. To compress this high-dimensional vector into a low-dimensional representation, learnable linear compression is used, combining mapping with low-dimensional subspace projection. Its calculation expression is as follows: ;in, Indicates the first The robot in the first The value snapshot summary generated in each decision cycle is a low-dimensional vector, with a dimension much smaller than that of the local value function. The dimension is used for transmission in channels with limited bandwidth; Indicates the first The robot in the first The local value function vector for each decision cycle is output by the robot's value network based on the current observation state. Each component corresponds to the value estimate of a candidate action in the action space. The compressed mapping weight matrix is ​​a learnable parameter. Its number of rows is the target dimension after compression, and its number of columns is the original dimension of the local value function. It is used to linearly map the high-dimensional value vector to a low-dimensional space. The compressed mapping bias vector is a learnable parameter whose dimension is equal to the compressed target dimension. It is used to add an offset after mapping to enhance expressive power. For projection to a lower-dimensional subspace This operation further constrains the linearly mapped vector to a predefined low-dimensional subspace. Within this scope, ensure that the compression result meets the dimensional constraints of transmission bandwidth; For a predefined low-dimensional subspace, the dimension is determined by the available bandwidth of the communication channel; the narrower the bandwidth, the smaller the subspace. The lower the dimension, the greater the degree of compression.

[0077] In the compression mapping and As learnable parameters, they are optimized by minimizing compression and reconstruction errors during the communication simulation phase of multi-robot collaborative training. This ensures that the compressed value snapshot summary retains, to the greatest extent possible, the key information in the original value function that has a decisive impact on action ranking and decision-making, while discarding redundant details that have little impact on decision-making; projection operation The introduction of this feature further ensures that the dimensions of the compression result are strictly controlled, avoiding mismatch of transmission protocols due to fluctuations in the dimensions of the compression mapping output.

[0078] The purpose of this step is to resolve the contradiction between limited communication bandwidth and the integrity of value information in multi-robot collaborative operations. By compressing the high-dimensional value function into a low-dimensional value snapshot summary, the core structural information of the value function is preserved while significantly reducing the amount of data transmitted, such as the relative ranking of the superiority and inferiority of actions and the distribution characteristics of high-value areas. This allows the receiver to reconstruct a sufficiently accurate value estimate based on the limited compressed information, providing an information foundation for subsequent consensus verification. Compared with simple compression methods such as direct truncation or random sampling, the learnable compression mapping can adaptively retain the most critical dimensional information for decision-making, achieving higher information fidelity under the same bandwidth constraints.

[0079] When transmitting a value snapshot digest via a limited bandwidth channel, the receiver performs integrity verification and decoding of the received data after generating the value snapshot digest. A robot will It communicates with its communication neighbor set through a limited bandwidth communication channel. All robots within the system are broadcast or directed to transmit data; due to bandwidth limitations, transmission delays, and potential packet loss risks in the communication channel, the receiver first performs an integrity check upon receiving the value snapshot summary;

[0080] Specifically, the recipient's first A robot checks the received value snapshot summary. Is the dimension of the low-dimensional subspace the same as the predefined low-dimensional subspace? If the dimensions are consistent, the transmission is considered abnormal; if the dimensions are inconsistent, the robot's value information is marked as unavailable in this decision-making cycle. If the dimensions are consistent, the transmission is considered successful, and the process proceeds to the subsequent consensus verification stage.

[0081] The purpose of this step is to: establish a reliable data transmission mechanism under communication constraints; in a multi-robot communication environment with limited bandwidth, packet loss and data corruption are common problems, and integrity verification can identify transmission anomalies in a timely manner, avoiding incorrect decisions based on corrupted data; for robots that fail to transmit, the system will automatically reduce the reliance weight on the robot's value information in subsequent conservative estimation modes, thereby ensuring that the system can still operate normally when some communication fails, reflecting the robust design of communication.

[0082] When the receiver performs consensus verification by comparing the received value snapshot summary with its local historical value estimate, after confirming the integrity of the transmitted data, the receiver... The robot will receive the first A snapshot summary of the value of a robot , and its own relationship with the first The local historical estimates of the value function of the first robot are compared, and the deviation between the two is calculated to determine the value of the second robot. Whether the perceived value of a robot aligns with the recipient's expectations can be calculated using the following formula: ;in, Indicates the first The consensus deviation between the value snapshot summary of the first robot and the recipient's local historical value estimate is a non-negative real number. The larger the value, the more significant the consensus deviation. The greater the deviation between the current perceived value of a robot and the historical expectations of the recipient; For the first A snapshot summary of value generated and transmitted by a robot; The first maintenance for the recipient The local historical value estimation vector of the robot, which is estimated by the receiver based on the data received within the past several decision-making periods. The value snapshot summary sequence of the first robot is maintained through exponential moving average or recursive update methods, reflecting the receiver's value of the first robot. Historical understanding of the value function of a robot; The local encoding function corresponding to the compression mapping has a structure similar to the compression mapping in the preceding steps. Similarly, it is used to estimate the high-dimensional historical value maintained by the recipient. Encode to the same low-dimensional subspace as the value snapshot summary. This allows the two to be compared within the same representation space; Representing vectors Norms are used to measure the distance between two low-dimensional vectors, compressing multi-dimensional biases into a single scalar.

[0083] The logic of this expression is that since the receiver cannot directly obtain the complete high-dimensional value function of the sender, it is necessary to make a comparison in the common low-dimensional representation space of both parties. The function ensures that the local historical value estimate and the received value snapshot summary are in the same representation space and dimension, making the bias calculation meaningful; if the... The robot's strategy has not changed significantly, and its current value snapshot summary should be highly consistent with the recipient's historical estimates, with minimal deviation. Approaching zero; if the first The robot's strategy has changed significantly due to environmental changes, its own drift, or other reasons, resulting in a large change in its value function and bias. This will increase significantly, indicating a divergence in the value perceptions of the two parties.

[0084] The purpose of this step is to establish a value consensus verification mechanism among multiple robots. Under limited communication conditions, robots cannot share complete observation and strategy information in real time, and value perception disagreements are a significant hidden danger leading to collaborative failure. Through consensus verification, the system can detect value perception disagreements with extremely low communication overhead, that is, only need to transmit low-dimensional summaries instead of the complete value function, and promptly discover collaborative decision-making conflicts that may be caused by information asymmetry, providing triggering conditions for subsequent conservative arbitration.

[0085] When the consensus deviation exceeds a preset threshold, a conservative estimation mode is activated to arbitrate the action. When outputting the final control command, the consensus deviation is obtained. Then, it is compared with a preset consensus threshold. Compare the results and select different action arbitration strategies based on the comparison results;

[0086] If the consensus deviation does not exceed the threshold, it indicates that the value perception is consistent, and the correction action output by the aforementioned two-layer drift suppression module is directly accepted; if the consensus deviation exceeds the threshold, it indicates that there is a significant divergence in value perception, and a conservative estimation mode is activated. Between the received remote value estimate and the local conservative value estimate, a conservative value is taken for action arbitration, and its calculation expression is as follows: ;in, Indicates the first The robot in the first Each decision cycle outputs a coordinated control command, which is the actual action performed by the robot in that decision cycle. For the first The robot's corrected action after double-layer drift suppression is directly accepted if the consensus deviation does not exceed the threshold. The consensus bias is calculated. The preset consensus threshold is a positive real number, which determines the sensitivity to triggering the conservative estimation mode. The smaller the value, the more sensitive the system is to value disagreements and the more easily it triggers a conservative mode. The larger the value, the greater the value that the system can tolerate, which triggers the conservative mode. The specific value can be determined by simulation training based on historical sample data. Indicates the candidate set of safety actions In this process, we search for actions that maximize the objective function, i.e., we select the optimal action from the set of safe actions after double-layer drift suppression. The set of corrected actions output by the aforementioned steps after double-layer drift suppression has passed the individual safety envelope constraint and cooperative consistency check. The action value estimate is reconstructed based on the received value snapshot summary, representing the current state based on the compressed value information transmitted remotely. Next action Long-term return expectations; This refers to the global or local observation status during the current decision-making cycle. This is a local conservative value estimate, indicating the first... The robot, based solely on its own local observations and historical experience, determines the actions... The value assessment is calculated through a local value network, and its estimation strategy adopts a conservative principle, that is, it tends to underestimate the value of actions under uncertain conditions; The logic behind the minimum value operation is to select the lower of the remote reconstruction value estimate and the local conservative value estimate, reflecting the conservative principle of selecting the more pessimistic estimate between the two.

[0087] The arbitration logic of this expression falls into two categories: the first category is consensus deviation. At this point, the remote value information is consistent with the local historical expectations, so no conservative intervention is needed. The correction action after double-layer drift suppression can be executed directly. The first scenario is maintaining the normal execution efficiency of the system; the second scenario is consensus deviation. At this point, a significant discrepancy emerges between the remote value information and the local expectation. The system cannot determine which value estimate is more reliable, therefore a conservative strategy is adopted: reconstructing the value remotely. with local conservative values Between these two values, the minimum value is taken, and the action that maximizes that minimum value is chosen. This maximum-minimum strategy ensures that the best return can still be obtained under the most pessimistic value estimate, and avoids overly aggressive behavior caused by differences in value perception.

[0088] The purpose of this step is twofold: First, under conditions of limited communication leading to incomplete information, a conservative estimation mechanism ensures the safety of action decisions. Second, when value consensus verification reveals discrepancies, the system does not blindly trust the value judgment of either party (remote or local), but instead takes a conservative lower bound of the two as the basis for decision-making, ensuring that even in the worst-case scenario, the action remains within a safe and controllable range. This conservative arbitration strategy effectively reduces the risk of collaborative conflicts caused by differing value perceptions in multi-robot collaborative scenarios with incomplete information. Simultaneously, when the consensus deviation does not exceed a threshold, the corrected action is directly adopted, avoiding unnecessary conservative intervention and ensuring that execution efficiency is not affected under normal communication conditions.

[0089] To ensure system-level consistency, the final control commands for each robot are verified globally for coordination. This is done after obtaining the final control commands for each robot. Subsequently, since the arbitration process for each robot is executed independently, there may be situations where the final commands of the robots do not meet the global coordination constraints. This step performs a global coordination verification on the final control commands of all robots, calculates the global coordination index, and determines whether the current set of control commands meets the system-level coordination requirements. The calculation expression is as follows: ;in, Indicates the first The global coordination index for each decision cycle, with a value range of [value range missing]. When the actions of all robots on the collaborative edges are completely consistent, Approaching 1, when the deviation in actions between robots is extremely large, Approaching 0; The total number of edges in the set of cooperative edges is used to normalize the average of the coordination of all cooperative edges. For the set of collaborative edges between robots, each edge Indicates the first The robot and the first There are direct collaborative relationships among the robots, and this set is determined by the task allocation structure and communication topology of the multi-robot system. For index pairs of collaborative edges, representing a path from the first edge... The robot to the first The collaborative relationships among the robots; For the obtained first The final control command for the robot; For the first The final control command for the robot; It represents the Euclidean distance between the final control commands of two robots, and measures the coordination deviation of their actions; To coordinate the bandwidth parameter, which is a positive real number, the tolerance level of the coordination evaluation is controlled. The larger the value, the higher the tolerance for movement deviations and the slower the decline in coordination indicators. The smaller the value, the more sensitive the person is to movement deviations and the faster their coordination indicators decline. The natural exponential function maps the squared non-negative distance to... The coordination score of the intervals is such that when the distance is zero, the score is 1 (perfect coordination), and as the distance increases, the score decreases according to the Gaussian decay law. This is a normalization factor to ensure that the global coordination index is not affected by changes in the number of collaborative edges.

[0090] This expression uses a Gaussian kernel function. The Gaussian kernel measures the degree of coordination between each pair of collaborative robots. Its physical meaning is as follows: when the control commands of two collaborative robots are completely identical, the Gaussian kernel output is 1, indicating perfect coordination; as the deviation between the two commands increases, the Gaussian kernel output decays exponentially, and the degree of coordination gradually decreases; when the deviation is much greater than... When the Gaussian kernel output approaches 0, it indicates severe incoordination; the average Gaussian kernel output of all cooperating edges is taken to obtain the global coordination index.

[0091] when Below the preset coordination lower limit hour, The preset global coordination lower limit threshold is a positive real number, and its value range is [value range missing]. If the current set of control commands does not meet the global coordination requirements, a global replanning process is triggered, which involves reverting to the policy network to re-perform decision-making and planning until control commands that meet the coordination requirements are output; when When the overall coordination is deemed to meet the requirements, control commands are formally issued to each robot for execution.

[0092] The purpose of this step is to serve as the final safety checkpoint for the entire system, ensuring that the control commands after independent arbitration by each robot still meet the requirements of coordination and consistency at the global level. Since the conservative arbitration of each robot is executed independently, different robots may choose different arbitration strategies due to their different consensus deviation states. That is, some robots directly adopt the corrective action, while some robots execute conservative arbitration. This heterogeneous arbitration result may lead to incoordination between robot actions. Global coordination verification, by performing Gaussian kernel weighted evaluation on the action deviations of all cooperative edges, can detect potential global coordination conflicts before final execution and ensure system-level safety by triggering a replanning mechanism.

[0093] After confirming the overall coordination indicators Not lower than the preset coordination lower limit Then, the system will send the final control commands to each robot. The data was officially distributed to the underlying motion controllers of each robot, driving each robot to perform corresponding actions in the physical environment; at the same time, each robot updated its local historical value estimate. The value snapshot summary received in this decision cycle is included in the historical sequence to provide a reference benchmark for consensus verification in the next decision cycle. At this point, the complete control process of this decision cycle ends, and the system enters the next decision cycle, starting from the first step of this embodiment.

[0094] The purpose of this step is to complete the entire closed-loop control process from perception, quantification, and suppression to arbitration, verification, and execution, ensuring that the output control commands reliably drive the robot to execute after passing through multiple layers of safety protection; at the same time, by updating historical value estimates, the system has the ability to continuously learn and adapt, continuously accumulate experience in continuous decision-making cycles, and improve the accuracy of consensus verification.

[0095] In this embodiment, a learnable compression mapping is used to compress the high-dimensional value function into a low-dimensional value snapshot summary. This significantly reduces communication bandwidth requirements while preserving the core decision-making information of the value function, resolving the contradiction between limited communication bandwidth and the integrity of value information in multi-robot collaborative operations. A consensus verification mechanism detects value perception discrepancies among multiple robots with extremely low communication overhead, promptly identifying potential collaborative decision-making conflicts even under incomplete information conditions. A maximum-minimum conservative arbitration strategy is employed to make decisions based on conservative lower bounds of remote and local estimates when value perception discrepancies arise, ensuring that actions remain safe and controllable even in the most pessimistic scenario, avoiding overreactions caused by information asymmetry. Aggressive behavior; through global coordination verification as the final safety checkpoint, it ensures that the independent arbitration results of each robot meet the coordination consistency at the global level and prevents coordination conflicts caused by heterogeneous arbitration; when the entire module is in normal communication, it directly adopts the corrective actions of the preceding module without introducing additional delays and conservatism. The conservatism mechanism is only activated when communication is abnormal or there is a value disagreement, realizing the adaptive switching of normal efficiency and abnormal safety; the above design effectively solves the technical problems of value perception disagreement, collaborative decision conflict and unreliable action caused by incomplete information in multi-robot collaborative linkage under communication-limited conditions. Under the constraint of limited bandwidth, it realizes safe, efficient and robust collaborative linkage control.

[0096] In the several embodiments provided by this invention, it should be understood that the disclosed system can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative; for example, the division of modules is merely a logical functional division, and in actual implementation, there may be other division methods.

[0097] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Depending on actual needs, some or all of the modules can be selected to achieve the purpose of this embodiment.

[0098] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or in a combination of hardware and software functional modules.

[0099] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the essential characteristics of the present invention.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An AI-based intelligent control system for robot collaborative operation, characterized in that, include: Nonstationarity quantification and environmental anchoring confidence map construction module: Collect the action probability distribution of each robot's policy network within a continuous decision cycle. By calculating the Wasserstein distance between the action probability distributions of adjacent decision cycles and obtaining the rate of change over time, the policy drift gradient of each robot is obtained. The policy drift gradients of multiple robots are spatially aggregated to construct a cooperative nonstationarity tensor. Based on the cooperative nonstationarity tensor, an environmental anchoring confidence map characterizing the reliability of the current environmental state of each robot is generated. Adaptive two-layer drift suppression module: Based on the generated environmental anchor confidence map, it performs two-layer collaborative drift suppression of feedforward prediction and feedback constraint: First, it uses the historical sequence of policy drift gradient to construct an exponentially weighted trend predictor to estimate the drift direction and magnitude of the next decision cycle. Then, it applies a feedforward pre-compensation force in the opposite direction to the original output action of the policy to weaken the drift trend from the source. Based on the environment anchor confidence map, a policy safety envelope that dynamically shrinks or expands with confidence is constructed. The action after feedforward pre-compensation is subjected to envelope boundary detection. If the boundary still exists, the policy bounce correction is triggered, and the action is mapped back to the safe area within the envelope. The corrected action set after double-layer drift suppression is output. Value consensus verification and action arbitration module: Based on the output set of corrected actions, when communication is limited, the local value function of each robot is compressed into a low-dimensional value snapshot summary and transmitted with limited bandwidth. The receiver performs consensus verification with the received value snapshot summary and the local historical value estimate. When the deviation exceeds the preset threshold, the conservative estimation mode is activated to arbitrate the action and output the final coordinated linkage control command.

2. The AI-based intelligent control system for robot collaboration according to claim 1, characterized in that, During the collaborative operation of multiple robots, each robot is equipped with an independent policy network. In each decision cycle, the policy network outputs an action probability distribution based on the current observation state. The system continuously collects the action probability distribution sequence of each robot in consecutive decision cycles to form the policy evolution time series of each robot.

3. The AI-based intelligent control system for robot collaboration according to claim 2, characterized in that, After obtaining the number of robots in the first Wasserstein distance over decision cycles Next, the rate of change of this distance between adjacent decision periods is calculated, i.e., the policy drift gradient, which is expressed as follows: ;in, Indicates the first The robot in the first The policy drift gradient over each decision cycle; For the calculated first The robot in the first Wasserstein distance over each decision cycle; For the first The robot in the first Wasserstein distance over each decision cycle; The time interval between adjacent decision cycles; For robot index variables, , This indicates the total number of robots participating in the collaborative operation.

4. The AI-based intelligent control system for robot collaboration according to claim 3, characterized in that, The drift gradients of each robot are coupled according to their communication topology to construct a cooperative nonstationary tensor, the calculation expression of which is as follows: ;in, Indicates the first The collaborative nonstationary tensor of each decision cycle; Indicates the first The set of communication neighbors of each robot; Indicates the first The robot and the first Interaction weights between robots; For the calculated first The robot in the first The policy drift gradient over each decision cycle; For the first The transpose of the policy drift gradient of a robot.

5. The AI-based intelligent control system for robot collaboration according to claim 4, characterized in that, The environmental anchoring confidence value that can be used independently by each robot is calculated using the following formula: ;in, Indicates the first The robot in the first Environmental anchoring confidence level for each decision cycle; is the confidence decay coefficient, which is a positive real number; Represents the cooperative nonstationary tensor The Frobenius norm; is a smoothing constant, and is a positive real number; For the calculated first The policy drift gradient of each robot; Indicates the first All decision-making cycles Take the maximum value among the policy drift gradients of each robot; It is a natural exponential function; The environmental anchoring confidence values ​​of all robots together constitute the environmental anchoring confidence map.

6. The AI-based intelligent control system for robot collaboration according to claim 5, characterized in that, After obtaining the predicted drift gradient, at the th Apply a feedforward pre-compensation force opposite to the predicted drift direction to the original output action of the robot's policy network. This pulls the policy output back before the drift trend has fully developed, thus weakening the cumulative effect of drift at the source.

7. The AI-based intelligent control system for robot collaboration according to claim 6, characterized in that, After completing the feedforward pre-compensation, the environmental anchoring confidence level of the preceding output is used to provide the first... Each robot constructs a policy safety envelope and determines the dynamic radius of the envelope. The policy safety envelope is a spherical safety region with the historical safe anchor point action as the center and the dynamic radius as the boundary. Any policy output action must fall within this spherical region to be considered safe.

8. The AI-based intelligent control system for robot collaboration according to claim 7, characterized in that, In the The decision cycle, the first The local value function of a robot is a high-dimensional vector whose dimension is equal to the number of discrete actions in the robot's action space or the number of discretized grids in the continuous action space. To compress this high-dimensional vector into a low-dimensional representation, learnable linear compression is used, which combines mapping with low-dimensional subspace projection.

9. The AI-based intelligent control system for robot collaboration according to claim 8, characterized in that, When the conservative estimation mode is activated to arbitrate the action and output the final control command, after obtaining the consensus deviation, it is compared with the preset consensus threshold, and different action arbitration strategies are selected based on the comparison result. If the consensus deviation does not exceed the threshold, the correction action output by the aforementioned two-layer drift suppression module is directly accepted; if the consensus deviation exceeds the threshold, the conservative estimation mode is activated, and the conservative value is used for action arbitration between the received remote value estimate and the local conservative value estimate.

10. The AI-based intelligent control system for robot collaboration according to claim 9, characterized in that, For the final control commands of all robots, perform global coordination verification, calculate global coordination index, and determine whether the current set of control commands meets the system-level coordination requirements.