Collaborative robot task optimization method based on multi-agent selective knowledge sharing

Abstract

The invention discloses a collaborative robot task optimization method based on multi-agent selective knowledge sharing, and belongs to the technical field of multi-agent reinforcement learning and industrial robot intelligent control. According to the method, for tasks such as splicing and carrying of multiple cooperative mechanical arms in an industrial manufacturing scene, only a success path and an effective adjustment strategy after failure are shared through a three-stage mechanism of knowledge source evaluation, content screening and personalized fusion; and correlation screening and weighted fusion are performed on the knowledge according to the task context and the local experience of the robot, so that blind simulation is avoided, and the personalized skills of the mechanical arms are reserved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multi-agent reinforcement learning, specifically to an optimization method for improving the decision-making efficiency and task completion quality of multiple collaborative robotic arms in multi-station assembly, handling, and other production processes through a selective knowledge-sharing mechanism. This method integrates technologies such as robot motion planning, real-time perception-decision control, distributed communication, and knowledge management, and can be widely applied in scenarios such as intelligent manufacturing, flexible production lines, human-robot collaboration, and the Industrial Internet. Background Technology

[0002] With the rapid development of intelligent manufacturing and flexible production lines, multiple collaborative robotic arms are being widely deployed in mobile phone assembly, engine block assembly, 3C material handling and sorting stations, placing higher demands on cycle time, consistency, and safe collaboration. Traditional methods often rely on offline programming or teach-and-play reproduction, lacking the ability to adapt online to fluctuations in incoming materials, fixture wear, and human-machine interaction interference.

[0003] In recent years, Multi-Agent Reinforcement Learning (MARL) has provided a self-learning collaborative decision-making framework for multi-robot systems. However, in complex production scenarios, MARL still faces challenges: the non-static environment causes the policies of each agent to change continuously, and the action space and combined states grow exponentially with the number of robotic arms. Relying solely on independent exploration results in slow convergence speed, making it difficult to meet industrial pace.

[0004] To improve learning efficiency, researchers have attempted to enable agents to share parameters, trajectories, or abstract strategies. However, existing "barrier-free" broadcast sharing also reveals significant drawbacks: the lack of assessment of the credibility and relevance of knowledge sources allows high-noise or locally optimal experiences to be propagated indiscriminately, leading to collisions, jamming, or abnormal assembly torque in collaborative robotic arms; excessive homogenization can erase the differences in individual skills of robotic arms such as fixture deformation compensation and visual bias, reducing the system's recovery capability during production line changes or tool wear; and the lack of budget control over sharing frequency and communication load can, in extreme cases, crowd out real-time control bus bandwidth and affect safety interlocks.

[0005] Therefore, existing MARL-driven collaborative robot systems urgently need a selective knowledge-sharing mechanism that can accurately identify knowledge value, limit ineffective propagation, and preserve local diversity, so as to achieve fast and robust policy convergence while ensuring pace and quality. Summary of the Invention

[0006] This invention aims to address the problems of low collaboration efficiency, poor policy generalization ability, and indiscriminate knowledge sharing in existing multi-agent reinforcement learning methods in complex industrial manufacturing scenarios. It proposes a collaborative robot task optimization method based on selective knowledge sharing among multiple agents. This method constructs a three-stage knowledge sharing mechanism of "dynamic request—trustworthy screening—personalized absorption," enabling the sharing of high-value experiences among multiple collaborative robotic arms while retaining the personalized strategies learned autonomously by each robot. This improves the overall completion efficiency and system stability in multi-station collaborative tasks.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] Step S1: Task environment initialization. Initialize the system for each robotic arm. Initialize the local policy network separately State value function Experience buffer and allocate the requested quota With the provision of quotas Control the frequency of knowledge exchange to ensure that the communication load is within a controllable range.

[0009] Step S2: Policy confidence calculation. Before each decision, the robotic arm calculates the probability distribution of all its executable actions based on the currently observed environmental state o. Calculate the standard deviation of the probability distribution. Construct a confidence index for normalized strategies The higher the confidence level, the more certain the robotic arm is about its own decisions, and the less it needs external advice.

[0010] Step S3: Dynamic request triggering mechanism. When the confidence level of a certain robotic arm... Less than the threshold And the number of times the current state has occurred in history. When the scarcity is low, the robotic arm operates with a probability inversely proportional to the "state access scarcity". Broadcast a knowledge request, sending information such as its own observations, the number of times those observations have been accessed, and the currently estimated value of the optimal action. And deduct one request quota accordingly. This is to prevent frequent requests for help.

[0011] Step S4: Trusted knowledge source screening. The robotic arm that receives the request... First, check your remaining quota. Secondly, verify whether you are more experienced or have made higher-value decisions in this situation. If the conditions are met, the candidate will be added to the advisor candidate set.

[0012] Step S5: Knowledge content relevance assessment and delivery. The candidate advisor calculates the entropy of their own strategy distribution as follows: Then construct the state-related suggestion probabilities: The more concentrated the action distribution and the more experienced the client, the higher the probability. If a random sample falls within this probability range, the consultant will send the optimal action pair to the requester. Worst-case scenario and the credibility of combining experience and strategy certainty The suggested message is as follows: After sending, it provides a quota. One deduction will be made accordingly.

[0013] Step S6: Personalized Knowledge Absorption. Based on the consultant's rating of each action as "good" or "bad," the requesting party assigns weights to positive and negative suggestions. Incorporate these weighted suggestions into your own action probability distribution Adjust decision-making tendencies in a gentle manner.

[0014] Step S7: Knowledge Interaction and Policy Iteration. Repeat steps S2–S6 until the collaborative task is completed or the policy converges.

[0015] This invention significantly reduces the spread of low-quality experience and prevents performance degradation caused by knowledge generalization by limiting the number of knowledge requests and provision, combined with knowledge source screening and content judgment strategies; through a positive and negative knowledge weighted absorption mechanism, each robotic arm retains the strategy expression ability adapted to its own task characteristics during shared learning.

[0016] This invention is particularly suitable for multi-station assembly, handling, locking and visual positioning scenarios. In system deployment, it can effectively improve the adaptability of production cycle time, the ability to cooperate and tolerate faults, and the efficiency of strategy migration, and has important practical application value and promotion prospects. Attached Figure Description

[0017] Figure 1 Method flowchart.

[0018] Figure 2 Information generation and reception module process.

[0019] Figure 3 The figure shows the experimental results comparing our method with the no-sharing (independent learning) method. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description is provided in conjunction with examples and accompanying drawings. However, the embodiments of this invention are not limited to these examples, and can be used to address similar problems.

[0021] Example 1:

[0022] This embodiment uses the collaborative operation of three robotic arms in a flexible assembly line as a case study to demonstrate how the selective knowledge sharing mechanism described in this invention can improve strategy learning efficiency and system stability. The method includes the following steps:

[0023] Step S1: Determine the multi-agent environment and perform initialization.

[0024] For each robotic arm Initialize local policy local policy network State value function Experience buffer During system initialization, a limited request quota is also allocated to each robotic arm. With the provision of quotas ;

[0025] All agents use the same neural policy model architecture, with state inputs including task stage, end-effector pose, and object position; the action space includes atomic actions such as grasping, moving, and releasing.

[0026] Step S2: Confidence Calculation and Request Judgment

[0027] Whenever the robotic arm To make a decision, the policy confidence level must first be calculated: ,in Let A represent the standard deviation of the probability distribution of actions in this state, and |A| represent the size of the action space.

[0028] If the confidence level is below the threshold And the current state Number of visits in the experience base Then, a knowledge request with the following probability will be triggered: The request packet format is: ;

[0029] Step S3: Consultant selection and recommendation generation;

[0030] Other robotic arms first check their remaining quota. Secondly, verify whether you are more experienced or have made higher-value decisions in this situation. If the conditions are met, the candidate will be added to the advisor candidate set.

[0031] Based on the frequency of their visits to this state and the degree of uncertainty in their action distribution, the candidate advisor dynamically calculates a probability of being "willing to offer advice". The more concentrated the action distribution and the more experience, the greater the probability.

[0032] If the random sample falls within this probability range, the consultant will send the optimal action pair to the requester. Worst-case scenario Based on credibility, the following suggestion package will be returned: After sending, it provides a quota. One deduction will be made accordingly.

[0033] Step S4: Knowledge Absorption and Strategy Revision

[0034] robotic arm For all suggestions, perform the following processing: If Then the action This is a positive suggestion; if If the suggestion is negative, then a negative suggestion is given; weights are assigned accordingly: And update the action distribution as follows: in The fusion temperature coefficient is set to 0.1.

[0035] Step S5: Task Execution and Strategy Iteration

[0036] Finally, based on the updated distribution Perform sampling and execution, and record the interaction trajectory. Write it into the experience buffer to participate in subsequent policy updates;

[0037] The entire process runs concurrently across three robotic arms, and all requests and advisory actions are subject to quota control to ensure that the system's communication bandwidth and computing resource consumption are stable and controllable.

[0038] Figure 3 The paper presents a comparison of the average rewards of the current embodiment at different training stages (100 rounds, 500 rounds, and 1000 rounds) in a collaborative exploration task. Experimental results show that the collaborative robot system based on multi-agent selective knowledge sharing proposed in this invention significantly outperforms traditional independent systems that do not share knowledge.

[0039] As can be seen from the above technical solutions, the present invention has the following advantages: it performs relevance screening and weighted fusion of knowledge, thereby avoiding blind imitation and preserving the personalized skills of each robotic arm.

Claims

1. A collaborative robot task optimization method based on multi-agent selective knowledge sharing, used for tasks such as assembly and handling by multiple collaborative robotic arms in industrial manufacturing scenarios, characterized in that... Includes the following steps: Step S1: Task environment initialization: Establish a local policy network and experience buffer for each robotic arm, and set the request quota and provide quota respectively; Step S2: Confidence Acquisition: Before each decision is made by the robotic arm, the probability distribution of all executable actions is calculated based on the current environmental state observed by the robotic arm. The standard deviation is then normalized to obtain the policy confidence Γ. o ; Step S3: Dynamic Request: When the confidence level is Γ o The value is less than the threshold, and the current state has occurred n times in history. visit (s) When the value is below the set upper limit, according to probability Broadcast knowledge requests, send knowledge requests, and deduct request quotas; Step S4: Knowledge Source Credibility Filtering: The robotic arm that receives the request enters the advisor candidate set when it provides sufficient quota and has more experience or higher value estimate in this state; Step S5: Content Relevance Screening and Sending: Candidate consultants dynamically calculate the probability of "willing to provide advice." If a random sample falls within this probability, the consultant will send the best action pair, worst action pair, and credibility level to the requester. The knowledge, and deduct the quota provided; Step S6: Knowledge Absorption at the Receiver: The requester absorbs knowledge based on the positive and negative knowledge weights w. n and w p These weighted suggestions are incorporated into the probability distribution of the action itself, and the sampled actions are executed and written into the experience buffer. Step S7: Repeat steps S2–S6 until the production task is completed.

2. The collaborative robot task optimization method based on multi-agent selective knowledge sharing according to claim 1, characterized in that: In step S1, a quota b is requested. ask This limits the number of times the robotic arm can actively request knowledge from the outside world throughout the entire training process; it provides a quota b. give This is used to limit the number of times the robotic arm can provide knowledge to other robotic arms.

3. The collaborative robot task optimization method based on multi-agent selective knowledge sharing according to claim 1, characterized in that: In step S2, based on the current environmental state o observed by the robotic arm, the probability distribution π(·∣o) of all its executable actions is calculated; the standard deviation σ of the probability distribution is also calculated. o , and according to Obtain the confidence level Γ of the normalization strategy o The higher the confidence level, the more certain the robotic arm is about its own decisions, and the less it needs external advice.

4. The collaborative robot task optimization method based on multi-agent selective knowledge sharing according to claim 1, characterized in that: In step S3, when the confidence level Γ of a certain robotic arm o Less than the threshold Γ th And the state it is in has occurred n times in history. visit When (s) is low, the robotic arm operates with a probability inversely proportional to the "state access scarcity". Broadcast a knowledge request, sending information such as its own observations, the number of times those observations have been accessed, and the currently estimated value of the optimal action. req =(o,n o maxQ(o,·)), and deduct the corresponding request quota b once. ask This is to prevent frequent and ineffective requests for help.

5. The collaborative robot task optimization method based on multi-agent selective knowledge sharing according to claim 1, characterized in that: In step S4, the other robotic arms that receive the request first check their remaining supply quota. Secondly, verify whether you are more experienced in this situation or have made higher decision-making value (maxQ). j (o i ,·)>maxQ i (o i If the conditions are met, the candidate will be added to the advisor candidate set.

6. The collaborative robot task optimization method based on multi-agent selective knowledge sharing according to claim 1, characterized in that: In step S5, the candidate advisor dynamically calculates a probability of "willing to offer advice" based on their access frequency in this state and the degree of uncertainty in their action distribution. The more concentrated the action distribution and the more experienced the client, the higher the probability. If a random sample falls within this probability, the consultant will send the optimal action pair (a) to the requester. b ,p b ), worst-case action pair (a) w ,p w ) and credibility knowledge After sending, it provides quota b give One deduction will be made accordingly.

7. The collaborative robot task optimization method based on multi-agent selective knowledge sharing according to claim 1, characterized in that: In step S6, the requesting party assigns weights to positive and negative suggestions based on the consultant's evaluation of each action as "good" or "bad". w p =1-w n Incorporate these weighted suggestions into your own action probability distribution Adjust decision-making tendencies in a gentle manner.

8. The collaborative robot task optimization method based on multi-agent selective knowledge sharing according to claim 1, characterized in that: The method is applicable to multi-station collaborative scenarios such as robot assembly, handling, screw fastening, and visual positioning. It can improve the assembly success rate and reduce the defect rate in the system.

Images