Multi-mechanical-arm cooperation method and device based on robot operation experience accumulation

Abstract

The invention relates to a multi-mechanical-arm cooperation method and device based on robot operation experience accumulation. Comprising a basic data layer, a middle processing layer, a feature layer, a public information layer and an interface layer, the current state serves as input, and robot experience actions serve as output; and meanwhile, interconnection and intercommunication with other mechanical arms are achieved based on vision / touch / force sense information, robot joint state information and the like, the motion path of each mechanical arm is planned, and target actions for executing tasks are output. And the multi-machine intelligent cooperation mechanism compares and evaluates the experience action and the target action according to the current state, outputs a trade-off factor, and takes the sum of the weighted experience action and the weighted target action as a robot execution action.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of robotics, and in particular to a multi-robot arm cooperation method and device based on robot work experience accumulation. BACKGROUND

[0002] Multi-robot arm cooperation refers to multiple robot arms working together to complete a complex operation task such as grasping, assembling, and carrying through coordinated control. However, multi-robot arm cooperation also faces many challenges, such as how to solve the collision problem between multiple robot arms, how to deal with unstructured working environments and uncertain task targets, and how to improve the coordination and adaptability of multiple robot arms. SUMMARY

[0003] To solve the problem of multi-robot arm cooperation in the prior art, embodiments of the present specification provide a multi-robot arm cooperation method and device based on robot work experience accumulation, which takes the current state as input and robot experience action as output; at the same time, based on visual / tactile / haptic information, robot joint state information, and other robot arms, interconnection and intercommunication are realized, the motion path of each robot arm is planned, and the target action for executing the task is output. The multi-robot intelligent cooperation mechanism compares and evaluates the experience action and the target action according to the current state, and outputs a weighting factor, and takes the sum of the weighted experience action and the weighted target action as the robot execution action.

[0004] The specific technical solutions of the embodiments of the present specification are as follows:

[0005] In one aspect, the embodiments of the present specification provide a multi-robot arm cooperation method based on robot work experience accumulation, the method comprising:

[0006] obtaining the task executed by a target robot and the environment perception information of the working environment in which the target robot is located;

[0007] determining the experience action of each execution time of the target robot executing the task from a pre-established experience action database according to the task and the environment perception information;

[0008] obtaining the visual data, tactile data, mechanical data, joint state information, and operation information of each robot arm of the target robot at the current execution time;

[0009] taking all the visual data, tactile data, mechanical data, joint state information, and operation information of all the robot arms of the target robot at the current execution time and the environment perception information as the state variable corresponding to the current execution time, and inputting the state variable into a pre-trained reinforcement learning model corresponding to the task for calculation to obtain the target action of the next execution time of the target robot;

[0010] The target action at the next execution time is compared and evaluated with the corresponding empirical action at the next execution time to obtain a trade-off factor;

[0011] The target action and the empirical action at the next execution time are weighted and calculated using the aforementioned trade-off factor to obtain the target robot's execution action at the next execution time.

[0012] The operation of each robotic arm of the target robot is controlled according to the execution action at the next execution moment.

[0013] Furthermore, the state variables are input into a pre-trained reinforcement learning model corresponding to the task for calculation, and the target action of the target robot at the next execution moment is further included as follows:

[0014] Calculate the Q-value between the state variable and all action variables in the action space of the reinforcement learning model;

[0015] The action variable with the largest Q value is taken as the target action.

[0016] Furthermore, the target action at the next execution time is compared and evaluated with the corresponding empirical action at the next execution time to obtain the trade-off factors, which further include:

[0017] Using the formula Δa=|a target -a exp Calculate the action difference between the target action and the experienced action, where Δa represents the action difference, a target Indicates the target action, a exp Indicates experiential actions;

[0018] Using the formula w = 1 / (1+e) α(Δa-β) Calculate the trade-off factor, where w represents the trade-off factor and α and β represent the adjustment parameters.

[0019] Furthermore, by using the aforementioned trade-off factor to weight the target action and the experienced action, the execution action of the target robot is further calculated, including:

[0020] Through formula Calculate the weighted target action, where, This indicates the weighted target action;

[0021] Through formula Calculate the weighted empirical actions, where, This represents the weighted empirical action;

[0022] Through formula Calculate the execution action, where a finalIndicates the execution of an action.

[0023] Furthermore, the visual data includes image data collected by the visual sensor at the end of the robotic arm, the tactile data includes contact force and contact position collected by the tactile sensor at the end of the robotic arm, the mechanical data includes joint torque and end force collected by the force sensor of the robotic arm, the joint state information includes the joint angle, velocity and acceleration of the robotic arm, and the operation information includes the operation type of the robotic arm.

[0024] Furthermore, the use of all robotic arm visual data, tactile data, mechanical data, joint state information, operational information, and environmental perception information at the current execution moment of the target robot as state variables corresponding to the current execution moment further includes:

[0025] For each robotic arm, the visual data, tactile data, mechanical data, joint state information, operation information and environmental perception information of the robotic arm are fused to obtain the state vector of the robotic arm;

[0026] The state vectors of all robotic arms are used to form the state variables.

[0027] Furthermore, the execution actions include motion commands and operation information for each robotic arm.

[0028] On the other hand, embodiments of this specification provide a multi-robotic arm collaborative device based on accumulated robot operation experience, the device comprising:

[0029] The task and environment information acquisition unit is used to acquire environmental perception information about the task being performed by the target robot and its working environment.

[0030] An experience action acquisition unit is used to determine the experience actions of the target robot at each execution moment of the task from a pre-established experience action database, based on the task and the environmental perception information.

[0031] The robotic arm status acquisition unit is used to acquire the visual data, tactile data, mechanical data, joint status information and operation information of each robotic arm at the current execution time of the target robot;

[0032] The target action calculation unit is used to take all the robotic arm visual data, tactile data, mechanical data, joint state information, operation information and environmental perception information at the current execution time of the target robot as the state variables corresponding to the current execution time, and input the state variables into the pre-trained reinforcement learning model corresponding to the task for calculation to obtain the target action of the target robot at the next execution time.

[0033] The trade-off factor calculation unit is used to compare and evaluate the target action at the next execution time with the corresponding empirical action at the next execution time to obtain the trade-off factor;

[0034] An execution action calculation unit is used to perform a weighted calculation on the target action and the empirical action at the next execution time using the weighting factor, so as to obtain the execution action of the target robot at the next execution time.

[0035] The control unit is used to control the operation of each robotic arm of the target robot in the next execution moment according to the execution action of the next execution moment.

[0036] On the other hand, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described method.

[0037] On the other hand, embodiments of this specification also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0038] Using the embodiments of this specification, the task to be performed by the target robot and the environmental perception information of the working environment in which the target robot is located are first obtained. Then, the experienced actions corresponding to both the task and the environmental perception information are obtained from the experienced action database. After obtaining the state information of each robotic arm at the current execution time, the state information is used as the state variable at the current execution time. The target action corresponding to the state variable at the current time is calculated using a reinforcement learning model. This target action is calculated based on the current state of each robotic arm and the task. Then, the target action and the experienced action at the next execution time are weighted and calculated to obtain the optimal execution action. The optimal execution action is then used to control each robotic arm of the target robot. By using the weighted calculation method of target action and experienced action in the embodiments of this specification, real-time and historical data can be integrated, and real-time environmental perception information and historical experienced actions can be considered simultaneously. This balances exploration (trying new actions) and utilization (using known effective actions). Through weighting, the system can maintain a certain degree of exploration while using existing experience to guide the selection of actions, thereby making decisions by combining the current state and past successful strategies.

[0039] By combining target actions and experienced actions, the system can make more robust decisions even when the environment changes or the perceived information is not entirely accurate. The embodiments in this specification can adjust the relative importance of target actions and experienced actions according to the specific circumstances of the current task and environment, thereby optimizing the robot's performance and improving the system's adaptability. By weighting target actions and experienced actions, the system can learn from experience and quickly adapt to new tasks, promoting the stability and efficiency of the learning process to achieve optimal operational results. Compared to real-time computation that relies entirely on reinforcement learning, which introduces certain risks, the weighted computation of target actions and experienced actions in the embodiments of this specification can smoothly transition from one policy to another, reducing the potential risks caused by policy mutations and improving conservatism. Furthermore, when computational resources are limited, the embodiments in this specification, by combining experienced actions, can significantly reduce the demand for computational resources and reduce computational resource consumption. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 The figure shown is a schematic diagram of an implementation system for a multi-robotic arm collaboration method based on the accumulation of robot operation experience in an embodiment of this specification;

[0042] Figure 2 The diagram shown is a flowchart illustrating a multi-robotic arm collaboration method based on accumulated robot operation experience, according to an embodiment of this specification.

[0043] Figure 3 The diagram shown is a flowchart illustrating the process of forming the state variable corresponding to the current execution moment in an embodiment of this specification.

[0044] Figure 4 The diagram shown is a flowchart illustrating how the state variables are input into a pre-trained reinforcement learning model corresponding to the task to calculate the target action of the target robot at the next execution moment, as described in this embodiment of the specification.

[0045] Figure 5 The diagram shown is a structural schematic of a multi-robotic arm collaborative device based on the accumulation of robot operation experience in an embodiment of this specification.

[0046] Figure 6 The diagram shown is a structural schematic of the computer device in an embodiment of this specification.

[0047] [Explanation of Figure Markers]:

[0048] 101. Terminal;

[0049] 102. Server;

[0050] 501. Task and Environmental Information Acquisition Unit;

[0051] 502. Experience Action Acquisition Unit;

[0052] 503. Robotic arm status acquisition unit;

[0053] 504. Target motion calculation unit;

[0054] 505. Trade-off factor calculation unit;

[0055] 506. Action Calculation Unit;

[0056] 507. Control Unit;

[0057] 602. Computer equipment;

[0058] 604, Processor;

[0059] 606. Memory;

[0060] 608. Drive mechanism;

[0061] 610. Input / output module;

[0062] 612. Input devices;

[0063] 614. Output devices;

[0064] 616. Presentation equipment;

[0065] 618. Graphical User Interface;

[0066] 620. Network interface;

[0067] 622. Communication link;

[0068] 624. Communication bus. Detailed Implementation

[0069] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this specification.

[0070] It should be noted that the terms "first," "second," etc., in the description, claims, and accompanying drawings of the embodiments herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0071] It should be noted that the acquisition, storage, use, and processing of data in the technical solutions of the embodiments of this specification all comply with the relevant provisions of national laws and regulations.

[0072] It should be noted that in the embodiments of this specification, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0073] like Figure 1 The diagram illustrates an implementation system for a multi-robotic arm collaboration method based on accumulated robot operation experience, as described in this specification. The system includes a terminal 101 and a server 102. The terminal 101 and server 102 can communicate via a network, which may include a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, and be connected to a website, user equipment (e.g., a computing device), and a backend system.

[0074] Terminal 101 can be multiple sensors with specific functions, used to collect visual data, tactile data, mechanical data, joint status information, operational information, and environmental perception information of the target robot's working environment from each robotic arm. The collected data is then sent to server 102. Furthermore, users can input tasks for the target robot to perform through terminal 101 to server 102. Server 102 calculates the target robot's actions based on the data sent by terminal 101 and controls the target robot accordingly.

[0075] Alternatively, server 102 may be a node of a cloud computing system (not shown in the figure), or each server may be a separate cloud computing system comprising multiple computers interconnected by a network and operating as a distributed processing system.

[0076] In addition, it should be noted that, Figure 1 The examples shown are merely one application environment provided by the embodiments in this specification. In practical applications, other application environments may also be included, and this specification does not impose any limitations.

[0077] To address the problems existing in the prior art, this specification provides a multi-arm collaborative method based on the accumulation of robot operation experience. The method takes the current state as input and the robot's experienced actions as output. Simultaneously, it achieves interconnection with other robotic arms based on visual / tactile / force information, robot joint state information, etc., plans the motion paths of each robotic arm, and outputs the target action for the task. The multi-arm intelligent collaboration mechanism compares and evaluates the experienced actions and the target actions based on the current state, and outputs a weighting factor. The sum of the weighted experienced actions and the weighted target actions is used as the robot's executed action.

[0078] Figure 2 The diagram illustrates a multi-arm collaborative method based on accumulated robot operation experience in an embodiment of this specification. The process of controlling each arm of the target robot is described in the diagram. The order of steps listed in the embodiment is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or accompanying drawings can be executed sequentially or in parallel.

[0079] Specific examples Figure 2 As shown, the method may include:

[0080] Step 201: Obtain the task being performed by the target robot and its environmental perception information of its working environment;

[0081] Step 202: Based on the task and the environmental perception information, determine the experienced actions of the target robot at each execution moment of the task from the pre-established experience action database;

[0082] Step 203: Obtain the visual data, tactile data, mechanical data, joint status information, and operation information of each robotic arm at the current execution time of the target robot;

[0083] Step 204: Take all the robotic arm visual data, tactile data, mechanical data, joint state information, operation information and environmental perception information at the current execution time of the target robot as the state variables corresponding to the current execution time, and input the state variables into the pre-trained reinforcement learning model corresponding to the task for calculation to obtain the target action of the target robot at the next execution time.

[0084] Step 205: Compare and evaluate the target action at the next execution time with the corresponding empirical action at the next execution time to obtain the trade-off factor;

[0085] Step 206: Use the aforementioned trade-off factor to perform a weighted calculation on the target action and the empirical action at the next execution time to obtain the target robot's execution action at the next execution time;

[0086] Step 207: Control the operation of each robotic arm of the target robot in the next execution moment according to the execution action of the next execution moment.

[0087] Using the embodiments of this specification, the task to be performed by the target robot and the environmental perception information of the working environment in which the target robot is located are first obtained. Then, the experience actions corresponding to both the task and the environmental perception information are obtained from the experience action database. After obtaining the state information of each robotic arm at the current execution time, the state information is used as the state variable at the current execution time. The target action corresponding to the state variable at the current time is calculated using a reinforcement learning model. The target action is calculated based on the current state and task of each robotic arm. Then, the target action and the experience action at the next execution time are weighted and calculated to obtain the optimal execution action. The optimal execution action is then used to control each robotic arm of the target robot.

[0088] It should be noted that the weight allocation in the weighted calculation needs to be designed according to the actual application scenario and objectives to ensure that the system can effectively combine real-time information and historical experience to achieve optimal decision-making and control.

[0089] In this embodiment of the specification, the time interval between two adjacent execution moments of the target robot performing the task can be preset by the operator. This embodiment divides the process of the target robot performing the task into multiple execution moments. First, experienced actions are acquired. Experienced actions are the motion commands and operation information of each robotic arm at each execution moment when the same target robot performs the same task in the same working environment. Then, a trained reinforcement learning model is used to calculate the target action for the next execution moment based on the state variables of the current execution moment. Then, the target action for the next moment and the experienced actions of the corresponding execution moments are weighted and calculated to obtain the final execution action for the next moment, thereby controlling each robotic arm of the target robot.

[0090] In the embodiments of this specification, the tasks performed by the target robot may include grasping tasks, moving tasks, placing tasks, etc. The target robot may be an automated robot on a production line with multiple robotic arms, such as a multi-robotic arm system for building module connection and installation processes, or an automobile assembly robot, etc. The embodiments of this specification do not impose any limitations.

[0091] Environmental perception information can be acquired through various sensors deployed in the work environment, including environmental status data such as environmental parameters such as temperature, humidity, and light, as well as the positions of objects and obstacles in the work environment.

[0092] The visual data includes image data collected by the visual sensor at the end of the robotic arm, including RGB images, depth images, point cloud data, etc. The tactile data includes contact force and contact position collected by the tactile sensor at the end of the robotic arm. The mechanical data includes joint torque and end force collected by the force sensor of the robotic arm. The joint state information includes the joint angle, velocity, and acceleration of the robotic arm. The operation information includes the operation type of the robotic arm, such as grasping, rotating, placing, etc.

[0093] In the embodiments of this specification, the experience action database can store multiple cases under different tasks and environmental perception information, including the experience actions of each robotic arm of the target robot at each execution moment when performing different tasks.

[0094] like Figure 3 As shown, the method of using all robotic arm visual data, tactile data, mechanical data, joint state information, operational information, and environmental perception information at the current execution moment of the target robot as state variables corresponding to the current execution moment further includes:

[0095] Step 301: For each robotic arm, fuse its visual data, tactile data, mechanical data, joint state information, operational information, and environmental perception information to obtain the state vector of the robotic arm;

[0096] Step 302: Form the state variables from the state vectors of all robotic arms.

[0097] In the embodiments of this specification, a simulated experimental environment can be constructed to train the robot on data from performing various tasks, thereby obtaining a trained reinforcement learning model.

[0098] For example, the state space of a reinforcement learning model may include the position and posture of the robotic arm (joint angles of each robotic arm, position and posture of the end effector, etc.), environmental state (position of objects in the working environment, position of obstacles, etc.), and perception data (real-time data from vision, touch, and force sensors). The action space may include joint actions (rotation angles or movement distances of each joint, etc.), end effector actions (movement path, speed, grasping or releasing actions of the end effector, etc.), and cooperative actions (adjustment of cooperative strategies among multiple robotic arms, such as task allocation and path planning, etc.).

[0099] The reward function of a reinforcement learning model can include task completion reward (a high reward is given after successfully completing the task), collision penalty (a penalty is given when the robotic arm collides with an obstacle or other robotic arm), efficiency reward (the shorter the task completion time, the higher the reward), energy consumption penalty (the higher the energy consumption, the greater the penalty), and path smoothness penalty (the smoother the action path, the higher the reward).

[0100] Termination conditions for reinforcement learning models can include task completion (successfully completing the specified task), collision occurrence (the robotic arm experiences a severe collision, causing the task to be unable to continue), time limit exceeding (the task execution time exceeds a preset threshold), and robotic arm failure (any robotic arm malfunctions, making it impossible to continue operation).

[0101] For example, the training process of a reinforcement learning model is as follows:

[0102] 1. Model initialization:

[0103] Neural network design: Choose a neural network architecture, such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0104] Parameter initialization: Randomly initialize the weight parameters of the neural network.

[0105] 2. Data Acquisition and Preprocessing:

[0106] Simulation environment construction: Build a simulation environment to simulate the collaborative operation of multiple robotic arms.

[0107] Data Acquisition: Collect a large number of state-action-reward-next state (s,a,r,s') samples in the simulation environment.

[0108] 3. Training process:

[0109] Experience replay: The collected data is stored in an experience pool, and small batches of data are randomly drawn from it for training to break the correlation between data.

[0110] Target network: Introduce a target network and update its parameters periodically to stabilize the training process.

[0111] Policy update: Based on the current state, select the optimal action (ε-greedy policy) and obtain the new state and reward by executing the action.

[0112] Gradient descent: Using gradient descent algorithms (such as Adam) to update neural network parameters, minimizing the Q-value or the loss of the policy function. Q-function

[0113] Periodic evaluation: Regularly evaluate model performance and adjust hyperparameters (learning rate, discount factor, etc.) to optimize training results.

[0114] 4. Model optimization:

[0115] Enhancement strategies such as dual-Q networks, priority experience replay, and A3C are introduced to improve the convergence speed and stability of the model.

[0116] Exploration-Exploitation Balance: A balance is achieved between exploring new strategies and utilizing existing strategies by adjusting the ε value (exploration probability).

[0117] Multi-agent reinforcement learning: In multi-robot collaborative operation scenarios, multi-agent reinforcement learning methods (such as MADDPG) are used to improve overall collaboration efficiency.

[0118] 5. Model Validation and Testing:

[0119] Simulation environment validation: Test the trained model in a simulation environment to evaluate its performance under different tasks and environments.

[0120] Real-world testing: Experiments were conducted in a real multi-robotic arm system to verify the model's practicality and robustness.

[0121] It should be noted that the training process of reinforcement learning models is common knowledge in the field, and will not be described in detail in the embodiments of this specification.

[0122] According to one embodiment of this specification, such as Figure 4 As shown, inputting the state variables into a pre-trained reinforcement learning model corresponding to the task for calculation, and obtaining the target action of the target robot at the next execution moment further includes:

[0123] Step 401: Calculate the Q-value between the state variable and all action variables in the action space of the reinforcement learning model;

[0124] Step 402: Select the action variable with the largest Q value as the target action.

[0125] In the embodiments of this specification, the target action, the experienced action, and the executed action all include motion instructions and operation information for each robotic arm, wherein the motion instructions include joint actions, end effector actions, cooperative actions, etc., and the operation information includes grasping, moving, placing, etc.

[0126] It should be noted that the computation process of reinforcement learning models is common knowledge in the field, and will not be described again in the embodiments of this specification.

[0127] According to one embodiment of this specification, the target action at the next execution time is compared and evaluated with the corresponding empirical action at the next execution time to obtain a trade-off factor, which further includes:

[0128] Using the formula Δa=|a target -a expCalculate the action difference between the target action and the experienced action, where Δa represents the action difference, a target Indicates the target action, a exp Indicates experiential actions;

[0129] Using the formula w = 1 / (1+e) α(Δa-β) Calculate the trade-off factor, where w represents the trade-off factor and α and β represent the adjustment parameters.

[0130] In the embodiments described in this specification, the adjustment parameters α and β can be empirical values, and the trade-off factor w is a number between (0,1).

[0131] Furthermore, by using the aforementioned trade-off factor to weight the target action and the experienced action, the execution action of the target robot is further calculated, including:

[0132] Through formula Calculate the weighted target action, where, This indicates the weighted target action;

[0133] Through formula Calculate the weighted empirical actions, where, This represents the weighted empirical action;

[0134] Through formula Calculate the execution action, where a final Indicates the execution of an action.

[0135] In the embodiments of this specification, if the empirical action a exp With target action If the difference in actions Δa between them is 0, then it indicates that the experienced action a exp With target action There is no difference between them; they are equal, based on the action a. final =(1-w)·a target +w·a exp It can be calculated that a final =a target =a exp .

[0136] By following the steps and formulas above, a robot behavior response strategy guided by experience and knowledge can be implemented, enabling multiple robots to work in parallel, autonomously learn the optimal control strategy, and complete distributed or centralized coordinated control of multiple robotic arms, thereby improving the intelligence and generalization ability of the robotic arms.

[0137] Based on the same inventive concept, embodiments of this specification also provide a multi-robotic arm collaborative device based on accumulated robot operation experience, such as... Figure 5 As shown, the device includes:

[0138] The task and environment information acquisition unit 501 is used to acquire the task being performed by the target robot and the environmental perception information of its working environment.

[0139] The experience action acquisition unit 502 is used to determine the experience actions of the target robot at each execution moment of the task from a pre-established experience action database based on the task and the environmental perception information.

[0140] The robotic arm state acquisition unit 503 is used to acquire the visual data, tactile data, mechanical data, joint state information and operation information of each robotic arm at the current execution time of the target robot;

[0141] The target action calculation unit 504 is used to take all the robotic arm visual data, tactile data, mechanical data, joint state information, operation information and environmental perception information at the current execution time of the target robot as the state variables corresponding to the current execution time, and input the state variables into the pre-trained reinforcement learning model corresponding to the task for calculation to obtain the target action of the target robot at the next execution time.

[0142] The trade-off factor calculation unit 505 is used to compare and evaluate the target action at the next execution time with the corresponding empirical action at the next execution time to obtain the trade-off factor;

[0143] The execution action calculation unit 506 is used to perform weighted calculation on the target action and the empirical action at the next execution time using the weighting factor to obtain the execution action of the target robot at the next execution time.

[0144] The control unit 507 is used to control the operation of each robotic arm of the target robot in the next execution moment according to the execution action of the next execution moment.

[0145] Furthermore, the state variables are input into a pre-trained reinforcement learning model corresponding to the task for calculation, and the target action of the target robot at the next execution moment is further included as follows:

[0146] Calculate the Q-value between the state variable and all action variables in the action space of the reinforcement learning model;

[0147] The action variable with the largest Q value is taken as the target action.

[0148] Furthermore, the target action at the next execution time is compared and evaluated with the corresponding empirical action at the next execution time to obtain the trade-off factors, which further include:

[0149] Using the formula Δa=|atarget -a exp Calculate the action difference between the target action and the experienced action, where Δa represents the action difference, a target Indicates the target action, a exp Indicates experiential actions;

[0150] Using the formula w = 1 / (1+e) α(Δa-β) Calculate the trade-off factor, where w represents the trade-off factor and α and β represent the adjustment parameters.

[0151] Furthermore, by using the aforementioned trade-off factor to weight the target action and the experienced action, the execution action of the target robot is further calculated, including:

[0152] Through formula Calculate the weighted target action, where, This indicates the weighted target action;

[0153] Through formula Calculate the weighted empirical actions, where, This represents the weighted empirical action;

[0154] Through formula Calculate the execution action, where a final Indicates the execution of an action.

[0155] Furthermore, the visual data includes image data collected by the visual sensor at the end of the robotic arm, the tactile data includes contact force and contact position collected by the tactile sensor at the end of the robotic arm, the mechanical data includes joint torque and end force collected by the force sensor of the robotic arm, the joint state information includes the joint angle, velocity and acceleration of the robotic arm, and the operation information includes the operation type of the robotic arm.

[0156] Furthermore, the use of all robotic arm visual data, tactile data, mechanical data, joint state information, operational information, and environmental perception information at the current execution moment of the target robot as state variables corresponding to the current execution moment further includes:

[0157] For each robotic arm, the visual data, tactile data, mechanical data, joint state information, operation information and environmental perception information of the robotic arm are fused to obtain the state vector of the robotic arm;

[0158] The state vectors of all robotic arms are used to form the state variables.

[0159] Furthermore, the execution actions include motion commands and operation information for each robotic arm.

[0160] Since the principle of the above-mentioned device in solving the problem is similar to that of the above-mentioned method, the implementation of the above-mentioned system can refer to the implementation of the above-mentioned method, and the repeated parts will not be described again.

[0161] like Figure 6 The diagram shown is a structural schematic of a computer device according to an embodiment of this specification. The computer device in this embodiment can run the feature processing platform of this embodiment.

[0162] Computer device 602 may include one or more processors 604, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. Computer device 602 may also include any memory 606 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, memory 606 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any storage resource can be used to store information using any technology.

[0163] Furthermore, any storage resource can provide volatile or non-volatile retention of information.

[0164] Furthermore, any storage resource can represent a fixed or removable component of the computer device 602. In one case, when the processor 604 executes associated instructions stored in any storage resource or combination of storage resources, the computer device 602 can perform any operation of the associated instructions. The computer device 602 also includes one or more drive mechanisms 608 for interacting with any storage resource, such as a hard disk drive system, an optical disk drive system, etc.

[0165] Computer device 602 may also include an input / output module 610 (I / O) for receiving various inputs (via input device 612) and providing various outputs (via output device 614). A specific output mechanism may include a presentation device 616 and an associated graphical user interface (GUI) 618. In other embodiments, the input / output module 610 (I / O), input device 612, and output device 614 may be omitted, and the device may function solely as a computer device within a network. Computer device 602 may also include one or more network interfaces 620 for exchanging data with other devices via one or more communication links 622. One or more communication buses 624 couple the components described above together.

[0166] Communication link 622 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 622 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0167] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0168] This specification also provides computer-readable instructions, wherein when a processor executes the instructions, the program therein causes the processor to perform the above-described method.

[0169] It should be understood that in the various embodiments of this specification, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0170] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the embodiments of this specification, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0171] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments in this specification.

[0172] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0173] In the embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0174] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described in this specification, depending on actual needs.

[0175] Furthermore, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0176] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this specification, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0177] This specification describes the principles and implementation methods of the embodiments using specific examples. The above descriptions of the embodiments are only for the purpose of helping to understand the methods and core ideas of the embodiments in this specification. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments in this specification. Therefore, the content of this specification should not be construed as a limitation on the embodiments in this specification.

Claims

1. A multi-robotic arm collaborative method based on the accumulation of robot operation experience, characterized in that, The method includes: Acquire environmental perception information about the tasks performed by the target robot and its working environment; Based on the task and the environmental perception information, the experienced actions of the target robot at each execution moment in performing the task are determined from a pre-established experience action database. The visual data, tactile data, mechanical data, joint state information, and operation information of each robotic arm at the current execution time of the target robot are obtained. The visual data, tactile data, mechanical data, joint state information, operation information, and environmental perception information of the target robot at the current execution moment are used as the state variables corresponding to the current execution moment. The state variables are then input into a pre-trained reinforcement learning model corresponding to the task for calculation to obtain the target action of the target robot at the next execution moment. The target action at the next execution time is compared and evaluated with the corresponding empirical action at the next execution time to obtain a trade-off factor; The target action and the empirical action at the next execution time are weighted and calculated using the aforementioned trade-off factor to obtain the target robot's execution action at the next execution time. The operation of each robotic arm of the target robot is controlled according to the execution action at the next execution moment.

2. The method according to claim 1, characterized in that, The state variables are input into a pre-trained reinforcement learning model corresponding to the task for calculation, and the target action of the target robot at the next execution moment is further included as follows: Calculate the Q-value between the state variable and all action variables in the action space of the reinforcement learning model; The action variable with the largest Q value is taken as the target action.

3. The method according to claim 1, characterized in that, The target action at the next execution time is compared and evaluated with the corresponding empirical action at the next execution time to obtain the trade-off factors, which further include: Using the formula Δa=|a target -a exp Calculate the action difference between the target action and the experienced action, where Δa represents the action difference, a target Indicates the target action, a exp Indicates experiential actions; Using the formula w = 1 / (1 + e) α(Δa-β) Calculate the tradeoff factor, where w represents the tradeoff factor and α and β represent the adjustment parameters.

4. The method according to claim 3, characterized in that, The weighted calculation of the target action and the experienced action using the aforementioned trade-off factor to obtain the execution action of the target robot further includes: Through formula Calculate the weighted target action, where, This indicates the weighted target action; Through formula Calculate the weighted empirical actions, where, This represents the weighted empirical action; Through formula Calculate the execution action, where a final Indicates the execution of an action.

5. The method according to claim 1, characterized in that, The visual data includes image data collected by the visual sensor at the end of the robotic arm; the tactile data includes contact force and contact position collected by the tactile sensor at the end of the robotic arm; the mechanical data includes joint torque and end force collected by the force sensor of the robotic arm; the joint state information includes the joint angle, velocity and acceleration of the robotic arm; and the operation information includes the operation type of the robotic arm.

6. The method according to claim 1, characterized in that, The target robot's current execution time includes all robotic arm visual data, tactile data, mechanical data, joint state information, operational information, and environmental perception information as state variables corresponding to the current execution time, further comprising: For each robotic arm, the visual data, tactile data, mechanical data, joint state information, operation information and environmental perception information of the robotic arm are fused to obtain the state vector of the robotic arm; The state vectors of all robotic arms are used to form the state variables.

7. The method according to claim 1, characterized in that, The actions performed include motion commands and operational information for each robotic arm.

8. A multi-robotic arm collaborative device based on the accumulation of robot operation experience, characterized in that, The device includes: The task and environment information acquisition unit is used to acquire environmental perception information about the task being performed by the target robot and its working environment. An experience action acquisition unit is used to determine the experience actions of the target robot at each execution moment of the task from a pre-established experience action database, based on the task and the environmental perception information. The robotic arm status acquisition unit is used to acquire the visual data, tactile data, mechanical data, joint status information and operation information of each robotic arm at the current execution time of the target robot; The target action calculation unit is used to take all the robotic arm visual data, tactile data, mechanical data, joint state information, operation information and environmental perception information at the current execution time of the target robot as the state variables corresponding to the current execution time, and input the state variables into the pre-trained reinforcement learning model corresponding to the task for calculation to obtain the target action of the target robot at the next execution time. The trade-off factor calculation unit is used to compare and evaluate the target action at the next execution time with the corresponding empirical action at the next execution time to obtain the trade-off factor; An execution action calculation unit is used to perform a weighted calculation on the target action and the empirical action at the next execution time using the weighting factor, so as to obtain the execution action of the target robot at the next execution time. The control unit is used to control the operation of each robotic arm of the target robot in the next execution moment according to the execution action of the next execution moment.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images