Multi-machine system number-real consistency error collaborative prediction method and system based on multi-agent reinforcement learning

Abstract

The invention discloses a multi-machine system number-real consistency error collaborative prediction method and system based on multi-agent reinforcement learning, and belongs to the technical field of multi-robot systems and artificial intelligence. According to the method, each robot is modeled as an intelligent agent, a centralized training distributed execution framework is adopted, and collaborative prediction of errors is achieved. Each agent collects own, neighbor and environment state information in real time, multi-source information integration is carried out through feature embedding and gating fusion, the influence of different information sources on errors is dynamically evaluated by adopting a layered attention mechanism, and propagation and coupling rules of the errors in a cluster are effectively modeled. In the training stage, the centralized commentator network is utilized to evaluate the global prediction performance, the cooperative optimization of each executor network is guided, and the prediction precision is improved. According to the method, active and collaborative prediction of the multi-machine system number-real consistency error is realized, and a key technical support is provided for error feed-forward compensation and system reliability guarantee.

Description

Technical Field

[0001] This invention belongs to the field of multi-robot systems and artificial intelligence technology, specifically relating to a collaborative prediction method and system for the consistency error of multi-robot systems based on multi-agent reinforcement learning. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, multi-robot systems are increasingly being used in complex scenarios such as warehousing and logistics, collaborative manufacturing, and environmental monitoring. Digital twin technology provides an effective means for simulating, testing, and optimizing multi-robot systems by constructing high-fidelity virtual models. However, during the deployment from the digital space to the physical world, due to individual model mismatch, environmental uncertainties, and the complex interactions and communications between agents, inconsistencies inevitably arise between the digital model and the physical entity. In multi-robot collaborative tasks, these errors originating from individual robots can form cluster-level "emergent" errors through dynamic coupling and propagation within the system, which can lead to task failure or even safety accidents in severe cases.

[0003] Currently, existing methods for handling data-real consistency errors mainly fall into three categories: First, methods based on high-fidelity modeling and state mapping focus on passive error monitoring and post-event tracing, lacking forward-looking prediction capabilities and struggling to establish accurate models covering all dynamic and nonlinear factors; second, error modeling methods based on data fusion and filtering, which can handle errors caused by known noise, but have limited predictive capabilities for unmodeled interactions and unstructured high-dimensional errors; and third, methods based on traditional machine learning, such as using recurrent neural networks to learn the temporal patterns of errors, but often flatten multi-source information, failing to effectively distinguish and utilize the inherent structure and relative importance of heterogeneous information such as the robot itself, neighbor interactions, and the environment, and relying on a large amount of labeled data, resulting in insufficient generalization ability.

[0004] Furthermore, existing reinforcement learning methods are mostly focused on learning control strategies that are robust to errors, rather than directly predicting the errors themselves, and lack explicit modeling and collaborative prediction mechanisms for the propagation of errors among agents.

[0005] Therefore, there is an urgent need in this field for an innovative method that can proactively and collaboratively predict the consistency error of data and reality from the perspective of the whole system, so as to achieve intelligent perception, forward warning and online compensation of error, thereby ensuring the reliable and efficient operation of multi-robot systems under complex collaborative tasks. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a collaborative prediction method and system for real-data consistency errors in multi-machine systems based on multi-agent reinforcement learning. Each robot is modeled as an independent agent, and within a framework of centralized training and distributed execution, they collectively learn a collaborative strategy capable of accurately predicting their own future real-data errors. The key lies in introducing an innovative hierarchical attention mechanism, enabling each agent not only to focus on its own state but also to intelligently and dynamically evaluate the behavior of neighboring agents and the potential impact of key environmental factors on its errors. This achieves a deep understanding and accurate prediction of the error propagation patterns at the cluster level.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A collaborative prediction method for real-data consistency error in multi-machine systems based on multi-agent reinforcement learning includes:

[0009] Step 1: Acquisition of multi-source state information: Each agent in the multi-machine system acquires its own state information, the state information of neighboring agents within its communication range, and the state information of environmental entities in real time;

[0010] Step 2, Multi-source state information processing: Feature embedding is performed on the acquired multi-source state information, the fusion weights of different information sources are dynamically adjusted through a gated fusion unit, and a hierarchical attention mechanism is used for weighted processing to generate fused feature representations;

[0011] Step 3, Error Collaborative Prediction: Each agent uses its independent agent network to output its own predicted value of future data consistency error based on the fused feature representation;

[0012] Step 4, Model Co-optimization: A centralized training distributed execution paradigm is adopted. The global performance of the predictions of all agents is evaluated using a centralized critic network. Based on the global performance and the difference between the predictions and the actual errors, the policy gradient is calculated, and the networks of each agent and the centralized critic network are updated in real time.

[0013] Furthermore, in step 1, the self-state information includes the agent's kinematic parameters, dynamic state, and task completion progress; the neighbor agent state information includes the real-time operating state of the neighbor agent; and the environmental entity state information includes the position and attitude attributes of static or dynamic obstacles in the environment and the task target.

[0014] Furthermore, the hierarchical attention mechanism in step 2 includes:

[0015] Self-attention processing: weights the agent's own state features to capture the correlation of key internal features;

[0016] Inter-agent attention processing: weighting the state features of neighboring agents to model cooperative relationships that have a strong coupling effect on cluster errors;

[0017] Entity attention processing: Weights the state features of environmental entities to capture the impact of changes in key environmental factors on errors.

[0018] Furthermore, in step 2, before performing hierarchical attention weighting processing, feature embedding is first performed on the multi-source state information, mapping it to a unified high-dimensional vector space; then, through the gating fusion unit, gating parameters are learned and applied to dynamically adjust the weights of the three embedded features from itself, neighboring agents, and environmental entities in the fusion process.

[0019] Furthermore, the model collaborative optimization in step 4 specifically includes:

[0020] A centralized network of critics receives fused feature representations from all agents and outputs a global value assessment.

[0021] A time-difference objective is generated based on the global value assessment and the instantaneous reward calculated from the difference between the prediction error and the actual error of each agent.

[0022] The loss is calculated based on the time difference objective, and the policy gradient is generated to update the parameters of the agent network and the centralized critic network for each agent.

[0023] Furthermore, step 4 also includes an attention-based experience replay mechanism, which assigns sampling priorities to samples in the experience replay pool based on the attention weights obtained in the hierarchical attention weighting process in step 2, and prioritizes the selection of samples with high attention weights for model training.

[0024] Furthermore, the instant reward is defined based on the negative quadratic norm between the prediction error value and the actual consistency error value.

[0025] On the other hand, the present invention provides a multi-machine system data-real consistency error collaborative prediction system based on multi-agent reinforcement learning, comprising:

[0026] The multi-source state information acquisition module enables each agent in a multi-machine system to acquire its own state information, the state information of neighboring agents within its communication range, and the state information of environmental entities in real time.

[0027] The multi-source state information processing module is used to embed features into the acquired multi-source state information. It dynamically adjusts the fusion weights of different information sources through a gated fusion unit and uses a hierarchical attention mechanism for weighted processing to generate fused feature representations.

[0028] The error collaborative prediction module enables each agent to use its independent agent network to output a predicted value of its own future data consistency error based on the fused feature representation.

[0029] The model collaborative optimization module is used to adopt a centralized training distributed execution paradigm, utilize a centralized critic network to evaluate the global performance of the predictions of all agents, calculate the policy gradient based on the global performance and the difference between the predictions and the actual errors, and update the agent networks and the centralized critic network in real time.

[0030] Thirdly, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for collaborative prediction of multi-machine system data consistency error based on multi-agent reinforcement learning.

[0031] Fourthly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned method for collaborative prediction of real-data consistency error in a multi-machine system based on multi-agent reinforcement learning.

[0032] The beneficial effects of this invention are as follows:

[0033] Achieving collaborative prediction capabilities at the cluster level: For the first time, multi-machine error prediction is constructed as a collaborative learning task, explicitly modeling the propagation and coupling laws of errors among agents, breaking through the limitations of traditional individual prediction, and improving prediction accuracy and consistency from the perspective of the whole system.

[0034] Enhancing the intelligence of information processing: Through a three-layer mechanism of self-attention, inter-agent attention, and entity attention, the agent can adaptively focus on key information sources, effectively extract error-related features from multi-source heterogeneous data, and enhance the model's representation and generalization capabilities in dynamic and complex scenarios.

[0035] Balancing global optimization and distributed efficiency: It adopts a centralized training and distributed execution architecture, which not only ensures that training converges to the system's optimal level through a global value network, but also achieves low communication overhead and high real-time performance through local computing in actual deployment.

[0036] Accelerate model convergence and training efficiency: Innovatively introduce an attention-oriented experience playback mechanism, which prioritizes the sampling of training samples based on attention weights, enabling the model to focus on high-value experiences, significantly improving learning speed and reducing computational resource consumption. Attached Figure Description

[0037] Figure 1This is a block diagram illustrating the principle of a collaborative prediction method for the real-data consistency error of a multi-machine system based on multi-agent reinforcement learning, as proposed in this invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0039] like Figure 1 The diagram shown illustrates the principle block diagram of a collaborative prediction method for real-data consistency errors in multi-machine systems based on multi-agent reinforcement learning, according to the present invention. The core of this technical solution lies in employing a centralized training and distributed execution (CTDE) multi-agent reinforcement learning framework. The system models each robot in the multi-robot system as an independent agent. Each agent contains a local actor network for distributed prediction of its own errors. During the training phase, a shared centralized critic network is introduced to evaluate and optimize the collaborative prediction performance of all agents from a global perspective. The method includes:

[0040] Step 1: Acquisition of multi-source state information and error modeling.

[0041] The digital-real fusion environment serves as the execution platform for this invention. It includes: (1) a physical environment: two real six-axis physical robotic arms (denoted as robot i and robot j) and their controllers; (2) a digital environment: a high-fidelity digital twin model constructed in the physical simulation engine, corresponding one-to-one with the physical robotic arms. The sources of digital-real consistency errors mainly include minor differences in motor response, sensor measurement noise, and random delays introduced by the communication network between robotic arms.

[0042] In each control cycle t, each robot in the multi-robot system (taking robot i as an example) acts as an intelligent agent. Simultaneously collect three types of status information.

[0043] Self-state information This includes its own kinematic parameters (position, velocity), dynamic state (joint torque), and the progress of the current task.

[0044] Neighbor status information : Obtain the real-time operating status of other robots (neighboring intelligent agents) within its interaction range through the communication network (key interaction information such as the end effector pose and speed of robot j).

[0045] Entity status information : Perceive key entity attributes in the task environment, such as the position and orientation of the object to be grasped, and the distribution of static and dynamic obstacles.

[0046] These real-time collected multi-source state information are simultaneously input into the physical robot system (each robot has an independent actuator network). It is a parameter of In the multilayer perceptron (MLP) and its corresponding digital twin model, the difference in their output states constitutes the robot's digital-real consistency error vector at the current moment. It is defined as: the difference vector between the actual output state of the physical robot i and its simulated output state in its digital twin model under the same control command (e.g., the position error of the end effector). (and attitude error). The error vectors of all robots are combined into a cluster-level global error matrix, which serves as a supervision signal for subsequent prediction model learning.

[0047] Step 2: Multi-source information embedding, fusion, and hierarchical attention weighting.

[0048] To handle state information from different sources and dimensions, this invention designs a refined information processing flow.

[0049] Feature embedding: First, the state information is embedded through an independent embedding network (such as a multilayer perceptron, MLP). Neighbor status information Entity status information These three types of heterogeneous information are mapped onto a unified high-dimensional vector space to obtain feature vectors. , , This makes them comparable.

[0050] Gated fusion: Subsequently, a learnable gated fusion unit is used to perform fusion on the three embedded feature vectors. , , Perform weighted fusion:

[0051] ,

[0052] Here, This represents vector concatenation. It is a learnable weight matrix. It is a sigmoid activation function, whose output gating value determines the contribution ratio of the three information sources in the final fused feature.

[0053] This gating unit dynamically adjusts the fusion ratio of different information sources through training, enabling it to adaptively determine which information is more critical based on the current task context.

[0054] Hierarchical attention processing: in the fused features Before being fed into the prediction network, information is refined through a uniquely designed three-layer attention mechanism:

[0055] Self-attention: Analyzes the internal correlations of the agent's own state characteristics to capture the key internal factors that lead to errors.

[0056] Inter-agent attention: Calculate the cross-attention weights between the agent and its neighboring agents' states, identify the cooperative partners that have the greatest impact on its own error, and thus efficiently model the propagation path of error in the cluster.

[0057] Entity attention: Dynamically assess the importance of different entities (such as targets and obstacles) in the environment to ensure that predictions can quickly adapt to environmental changes.

[0058] After this layered processing, the system generates a rich, focused, and highly condensed fusion feature representation. .

[0059] Step 3: Error Co-prediction Based on MARL.

[0060] This invention employs an industry-leading multi-agent reinforcement learning paradigm that combines centralized training with distributed execution:

[0061] Distributed predictive execution (Actor): Each agent has its own independent actor network (ActorNetwork). The network receives the fused feature representation generated in step 2. It also outputs a specific predicted value for the future data consistency error. In actual deployment, each agent can independently complete predictions relying only on local observations and limited communication.

[0062] Step 4: Model co-optimization.

[0063] During the training phase, a shared, global, centralized critic network is introduced. This network receives state and prediction information from all agents in the cluster, thus providing an overall value assessment of the cluster's collaborative prediction performance. This global perspective avoids the suboptimal problem of optimizing only individuals at the expense of the collective interest, ensuring that the prediction strategy converges in the direction of improving the overall prediction accuracy of the cluster. Specifically, this includes:

[0064] Reward Calculation and Experience Storage: Calculate instant rewards for each agent The immediate reward is designed as the negative quadratic norm between the prediction error and the true error, which directly incentivizes prediction accuracy.

[0065] ,

[0066] This value is negative; the smaller its absolute value, the more accurate the prediction and the higher the reward. Finally, the complete information of this interaction will be provided. It is stored as an experience sample in the experience replay pool.

[0067] Attention-Driven Experience Replay: To accelerate training convergence, this invention introduces an attention-aware experience replay mechanism. In the experience replay pool, different sampling priorities are assigned to historical interaction data based on the attention weights calculated by the hierarchical attention mechanism in step 2. Samples deemed more important and information-rich by the attention mechanism will be sampled more frequently for training, thereby significantly improving learning efficiency.

[0068] Centralized Commentator Network Update: Centralized Commentator Network Receive the set of feature representations of all agents at time t Output global value assessment By calculating the time difference (TD) target :

[0069] ,

[0070] in, It is a discount factor. This represents the cluster's joint state at the next time step. Then, by minimizing the loss function... To update the value network:

[0071] .

[0072] Distributed Execution Network Update: Using Advantage Functions To calculate the policy gradient, and thereby update the agent network parameters for each agent. The advantage function reflects how much better the current forecast is than the average expectation:

[0073] ,

[0074] Policy gradient The calculation formula is as follows, and gradient ascent is used to optimize the agent's policy, enabling the agent to learn to output more accurate error predictions:

[0075] .

[0076] In actual operation, the system executes steps 1-3 in a loop. All agents acquire multi-source information in real time, generate feature representations through the information processing module, and then output error predictions from their respective agent networks. These high-precision predictions can be used in a real-time feedforward compensation controller to correct control commands, thereby proactively canceling errors before they actually occur, significantly improving the task success rate and operational reliability of the multi-robot system.

[0077] On the other hand, the present invention provides a multi-machine system for collaborative prediction of real-data consistency error based on multi-agent reinforcement learning, wherein each module is capable of executing the steps of the aforementioned method, specifically including:

[0078] The multi-source state information acquisition module enables each agent in a multi-machine system to acquire its own state information, the state information of neighboring agents within its communication range, and the state information of environmental entities in real time.

[0079] The multi-source state information processing module is used to embed features into the acquired multi-source state information. It dynamically adjusts the fusion weights of different information sources through a gated fusion unit and uses a hierarchical attention mechanism for weighted processing to generate fused feature representations.

[0080] The error collaborative prediction module enables each agent to use its independent agent network to output a predicted value of its own future data consistency error based on the fused feature representation.

[0081] The model collaborative optimization module is used to adopt a centralized training and distributed execution paradigm, utilize a centralized critic network to evaluate the global performance of the predictions of all agents, calculate the policy gradient based on the global performance and the difference between the predictions and the actual errors, and update the networks of each agent and the centralized critic in real time.

[0082] Thirdly, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for collaborative prediction of multi-machine system data consistency error based on multi-agent reinforcement learning.

[0083] Fourthly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned method for collaborative prediction of real-data consistency error in a multi-machine system based on multi-agent reinforcement learning.

[0084] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A collaborative prediction method for real-data consistency error in multi-machine systems based on multi-agent reinforcement learning, characterized in that, include: Step 1: Acquisition of multi-source state information: Each agent in the multi-machine system acquires its own state information, the state information of neighboring agents within its communication range, and the state information of environmental entities in real time; Step 2, Multi-source state information processing: Feature embedding is performed on the acquired multi-source state information, the fusion weights of different information sources are dynamically adjusted through a gated fusion unit, and a hierarchical attention mechanism is used for weighted processing to generate fused feature representations; Step 3, Error Collaborative Prediction: Each agent uses its independent agent network to output its own predicted value of future data consistency error based on the fused feature representation; Step 4, Model Co-optimization: A centralized training distributed execution paradigm is adopted. The global performance of the predictions of all agents is evaluated using a centralized critic network. Based on the global performance and the difference between the predictions and the actual errors, the policy gradient is calculated, and the networks of each agent and the centralized critic network are updated in real time.

2. The method for collaborative prediction of real-data consistency error in a multi-machine system based on multi-agent reinforcement learning according to claim 1, characterized in that, In step 1, the self-state information includes the agent's kinematic parameters, dynamic state, and task completion progress; the neighbor agent state information includes the real-time operating state of the neighbor agent; and the environmental entity state information includes the position and attitude attributes of static or dynamic obstacles in the environment and the task target.

3. The method for collaborative prediction of real-data consistency error in a multi-machine system based on multi-agent reinforcement learning according to claim 1, characterized in that, The hierarchical attention mechanism in step 2 includes: Self-attention processing: weights the agent's own state features to capture the correlation of key internal features; Inter-agent attention processing: weighting the state features of neighboring agents to model cooperative relationships that have a strong coupling effect on cluster errors; Entity attention processing: Weights the state features of environmental entities to capture the impact of changes in key environmental factors on errors.

4. The method for collaborative prediction of real-data consistency error in a multi-machine system based on multi-agent reinforcement learning according to claim 3, characterized in that, In step 2, before performing hierarchical attention weighting processing, feature embedding is performed on the multi-source state information and mapped to a unified high-dimensional vector space. Then, through the gating fusion unit, gating parameters are learned and applied to dynamically adjust the weights of the three embedded features from itself, neighboring agents and environmental entities in the fusion process.

5. The method for collaborative prediction of real-data consistency error in a multi-machine system based on multi-agent reinforcement learning according to claim 1, characterized in that, The model collaborative optimization in step 4 specifically includes: A centralized network of critics receives fused feature representations from all agents and outputs a global value assessment. A time-difference objective is generated based on the global value assessment and the instantaneous reward calculated from the difference between the prediction error and the actual error of each agent. The loss is calculated based on the time difference objective, and the policy gradient is generated to update the parameters of the agent network and the centralized critic network for each agent.

6. The method for collaborative prediction of real-data consistency error in a multi-machine system based on multi-agent reinforcement learning according to claim 5, characterized in that, Step 4 also includes an attention-based experience replay mechanism, which assigns sampling priorities to samples in the experience replay pool based on the attention weights obtained in the hierarchical attention weighting process in step 2, and prioritizes samples with high attention weights for model training.

7. The method for collaborative prediction of real-data consistency error in a multi-machine system based on multi-agent reinforcement learning according to claim 5, characterized in that, The instant reward is defined based on the negative quadratic norm between the prediction error value and the actual data consistency error value.

8. A multi-machine system for collaborative prediction of real-data consistency error based on multi-agent reinforcement learning, characterized in that, include: The multi-source state information acquisition module enables each agent in a multi-machine system to acquire its own state information, the state information of neighboring agents within its communication range, and the state information of environmental entities in real time. The multi-source state information processing module is used to embed features into the acquired multi-source state information. It dynamically adjusts the fusion weights of different information sources through a gated fusion unit and uses a hierarchical attention mechanism for weighted processing to generate fused feature representations. The error collaborative prediction module enables each agent to use its independent agent network to output a predicted value of its own future data consistency error based on the fused feature representation. The model collaborative optimization module is used to adopt a centralized training distributed execution paradigm, utilize a centralized critic network to evaluate the global performance of the predictions of all agents, calculate the policy gradient based on the global performance and the difference between the predictions and the actual errors, and update the agent networks and the centralized critic network in real time.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when one or more programs are executed by the one or more processors, the one or more processors implement the multi-machine system data consistency error collaborative prediction method based on multi-agent reinforcement learning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, enable the processor to implement the multi-machine system data consistency error collaborative prediction method based on multi-agent reinforcement learning as described in any one of claims 1-7.

Images