Multi-robotic arm collaboration method and apparatus based on accumulated operational experience of robots

Abstract

The present application relates to a multi-robotic arm collaboration method and apparatus based on accumulated operational experience of robots. The apparatus comprises: a basic data layer, an intermediate processing layer, a feature layer, a common information layer, and an interface layer. The current state is used as an input, and a robot experience action is used as an output; and at the same time, interconnection and intercommunication with other robotic arms are implemented on the basis of visual / tactile / force information, robot joint state information, etc., a motion path of each robotic arm is planned, and a target action for executing a task is output. An intelligent multi-robotic arm collaboration mechanism performs comparison and evaluation on an experience action and a target action according to the current state, and outputs a weighting factor to use the sum of a weighted experience action and a weighted target action as an action to be executed by a robot.

Description

Multi-robot arm cooperation method and device based on robot work experience accumulation TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of robots, and in particular to a multi-robot arm cooperation method and device based on robot work experience accumulation. BACKGROUND

[0002] Multi-robot arm cooperative work refers to multiple robot arms completing a complex operation task such as grasping, assembling, and carrying through coordinated control.

[0003] SUMMARY

[0004] To solve the problem of multi-robot arm cooperation in the prior art, embodiments of the present specification provide a multi-robot arm cooperation method and device based on robot work experience accumulation, which takes the current state as input and robot experience action as output; at the same time, based on visual / tactile / haptic information, robot joint state information, and other robot arms, interconnection and intercommunication are realized, the motion path of each robot arm is planned, and the target action of executing the task is output. The multi-robot intelligent cooperation mechanism compares and evaluates the experience action and the target action according to the current state, and outputs a weighting factor, so as to take the sum of the weighted experience action and the weighted target action as the robot execution action.

[0005] The specific technical solutions of the embodiments of the present specification are as follows:

[0006] In one aspect, the embodiments of the present specification provide a multi-robot arm cooperation method based on robot work experience accumulation, which comprises:

[0007] Obtaining the task executed by a target robot and the environment perception information of the working environment in which the target robot is located;

[0008] Determining the experience action of each execution time of the target robot executing the task from a pre-established experience action database according to the task and the environment perception information;

[0009] Obtaining the visual data, tactile data, mechanical data, joint state information, and operation information of each robot arm of the target robot at the current execution time;

[0010] all the robot arms vision data, tactile data, mechanical data, joint state information, operation information and the environment perception information of the target robot at the current execution time are taken as state variables corresponding to the current execution time, and the state variables are input into the pre-trained reinforcement learning model corresponding to the task for calculation to obtain the target action of the target robot at the next execution time;

[0011] The target action at the next execution time and the experience action corresponding to the next execution time are compared and evaluated to obtain a weighting factor;

[0012] The target action at the next execution time and the experience action at the next execution time are weighted and calculated by using the weighting factor to obtain the execution action of the target robot at the next execution time;

[0013] According to the execution action at the next execution time, the work of each robot arm of the target robot at the next execution time is controlled.

[0014] Further, the state variables are input into the pre-trained reinforcement learning model corresponding to the task for calculation to obtain the target action of the target robot at the next execution time, and the method further comprises:

[0015] The Q value between the state variable and all action variables in the action space of the reinforcement learning model is calculated;

[0016] The action variable with the maximum Q value is taken as the target action.

[0017] Further, the target action at the next execution time and the experience action corresponding to the next execution time are compared and evaluated to obtain a weighting factor, and the method further comprises:

[0018] The action difference between the target action and the experience action is calculated by the formula Δα = |α target -α exp |, wherein Δα represents the action difference, α target represents the target action, and α exp represents the experience action;

[0019] The weighting factor is calculated by the formula w = 1 / (1+e α(Δa-β) ), wherein w represents the weighting factor, and α and β represent adjustment parameters.

[0020] Further, the target action and the experience action are weighted and calculated by using the weighting factor to obtain the execution action of the target robot, and the method further comprises:

[0021] The weighted target action is calculated by the formula , wherein denotes the weighted target action;

[0022] The weighted experience action is calculated by the formula The execution action is calculated by the formula, wherein α denotes the weighted experience action;

[0023] The weighted experience action is calculated by the formula The execution action is calculated by the formula, wherein α final denotes the execution action.

[0024] Further, the visual data includes image data collected by a visual sensor at the end of the robot arm, the tactile data includes contact force and contact position collected by a tactile sensor at the end of the robot arm, the mechanical data includes joint torque and end force collected by a force sensor of the robot arm, the joint state information includes joint angle, velocity and acceleration of the robot arm, and the operation information includes operation type of the robot arm.

[0025] Further, all of the visual data, tactile data, mechanical data, joint state information, operation information and environment perception information of the target robot at the current execution time are further included as state variables corresponding to the current execution time.

[0026] For each robot arm, the visual data, tactile data, mechanical data, joint state information, operation information and environment perception information of the robot arm are fused to obtain a state vector of the robot arm.

[0027] The state vectors of all robot arms form the state variables.

[0028] Further, the execution action includes motion instructions and operation information of each robot arm.

[0029] In another aspect, the embodiments of the present specification provide a multi-robot arm cooperation device based on robot work experience accumulation, the device comprising:

[0030] A task and environment information acquisition unit is configured to acquire a task executed by a target robot and environment perception information of a working environment in which the target robot is located;

[0031] An experience action acquisition unit is configured to determine an experience action of each execution time of the target robot executing the task from a pre-established experience action database according to the task and the environment perception information;

[0032] A robot arm state acquisition unit is configured to acquire visual data, tactile data, mechanical data, joint state information and operation information of each robot arm at a current execution time of the target robot;

[0033] a target action calculation unit configured to input all of the visual data, the tactile data, the mechanical data, the joint state information, the operation information of the robot arms of the target robot at a current execution time, and the environment perception information corresponding to the current execution time as state variables corresponding to the current execution time into a pre-trained reinforcement learning model corresponding to the task, and to calculate the target action of a next execution time of the target robot;

[0034] a trade-off factor calculation unit configured to compare and evaluate the target action of the next execution time and the experience action corresponding to the next execution time to obtain a trade-off factor;

[0035] an execution action calculation unit configured to perform weighted calculation on the target action of the next execution time and the experience action of the next execution time by using the trade-off factor to obtain an execution action of the next execution time of the target robot;

[0036] a control unit configured to control the robot arms of the target robot to work at the next execution time according to the execution action of the next execution time.

[0037] In another aspect, the embodiments of the present specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory, and the processor implements the above method when executing the computer program.

[0038] In another aspect, the embodiments of the present specification also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0039] According to the embodiments of the present specification, the task performed by the target robot and the environment perception information of the working environment of the target robot are first obtained, and then the experience action corresponding to the task and the environment perception information is obtained from the experience action database. After the state information of each robot arm at the current execution time is obtained, the state information is taken as the state variable of the current execution time, and the reinforcement learning model is used to calculate the target action of the next time corresponding to the state variable of the current time. The target action is calculated according to the current state of each robot arm and the task. Then, the target action and the experience action of the next execution time are weighted and calculated to obtain the optimal execution action, so as to control each robot arm of the target robot by using the optimal execution action. Through the method of weighting and calculating the target action and the experience action in the embodiments of the present specification, real-time and historical data can be fused, and real-time environment perception information and historical experience actions are considered. Exploration (trying new actions) and utilization (using known effective actions) are balanced. Through weighting, the system can maintain a certain degree of exploration while utilizing existing experience to guide the selection of actions, so as to make decisions by combining the current state and past successful strategies.

[0040] By combining the target action and the experience action, the system can make more robust decisions even if the environment changes or the perception information is not completely accurate. The embodiments of the present specification can adjust the relative importance of the target action and the experience action according to the specific circumstances of the current task and environment, thereby optimizing the performance of the robot and improving the adaptability of the system. By weighting the target action and the experience action, the system can learn from experience and quickly adapt to new tasks, promote the stability and efficiency of the learning process, and achieve the best operation effect. Compared with relying entirely on real-time calculation of reinforcement learning, since the reinforcement learning calculation will introduce a certain risk, the embodiments of the present specification perform weighted calculation on the target action and the experience action, which can smoothly transition from one policy to another, reduce the potential risk caused by policy mutation, and improve the conservatism. In addition, in the case of limited computing resources, the embodiments of the present specification combine the experience action to greatly reduce the demand for computing resources and reduce computing resource consumption. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief descriptions will be given below for the drawings needed to be used in the embodiments or prior art descriptions.

[0042] FIG. 1 shows an embodiment of a multi-robot arm cooperation method based on robot work experience accumulation in the present specification;

[0043] FIG. 2 shows a flowchart of a multi-robot arm cooperation method based on robot work experience accumulation in an embodiment of the present specification;

[0044] FIG. 3 shows a flowchart of forming a state variable corresponding to a current execution time in an embodiment of the present specification;

[0045] FIG. 4 shows a flowchart of inputting the state variable into a pre-trained reinforcement learning model corresponding to the task for calculation to obtain a target action of the target robot at the next execution time in an embodiment of the present specification;

[0046] FIG. 5 shows a structural diagram of a multi-robot arm cooperation device based on robot work experience accumulation in an embodiment of the present specification;

[0047] FIG. 6 shows a structural diagram of a computer device in an embodiment of the present specification.

[0048]

Explanation of reference numerals

[0049] 101, terminal; 102, server; 501, task and environment information acquisition unit; 502, experience action acquisition unit; 503, mechanical arm state acquisition unit; 504, target action calculation unit; 505, trade-off factor calculation unit; 506, execution action calculation unit; 507, control unit; 602, computer device; 604, processor; 606, memory; 608, driving mechanism; 610, input / output module; 612, input device; 614, output device; 616, presentation device; 618, graphical user interface; 620, network interface; 622, communication link; 624, communication bus. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

[0051] As shown in FIG. 1, it is an implementation system schematic diagram of a multi-robot arm cooperation method based on robot work experience accumulation in the embodiments of the present application, including a terminal 101 and a server 102. The terminal 101 and the server 102 can communicate through a network, which can include a local area network (LAN), a wide area network (WAN), the Internet or a combination thereof, and be connected to a website, a user device (such as a computer device) and a backend system.

[0052] The terminal 101 can be a plurality of sensors with specific functions, used to collect visual data, tactile data, mechanical data, joint state information, operation information and environment perception information of the working environment of the target robot of each mechanical arm of the target robot, and send the collected data to the server 102. In addition, the user can also input the task executed by the target robot to the server 102 through the terminal 101. The server 102 calculates the execution action of the target robot according to the data sent by the terminal 101, and controls the target robot according to the execution action.

[0053] Optionally, the server 102 can be a node (not shown in the figure) of a cloud computing system, or each server can be a separate cloud computing system, including a plurality of computers interconnected by a network and working as a distributed processing system.

[0054] In addition, it should be noted that the application environment shown in FIG. 1 is only one application environment provided by the embodiments of the present application, and other application environments can also be included in actual application, which is not limited by the present application.

[0055] To solve the problems in the prior art, the embodiment of the specification provides a multi-robot arm cooperation method based on robot work experience accumulation, taking a current state as input and taking robot experience action as output; meanwhile, based on visual / tactile / haptic information, robot joint state information and other information, interconnection and intercommunication with other robot arms are realized, motion paths of the robot arms are planned, and target actions for executing tasks are output. The multi-robot intelligent cooperation mechanism compares and evaluates the experience action and the target action according to the current state, and outputs a weighting factor, so as to take the sum of the weighted experience action and the weighted target action as the robot execution action.

[0056] FIG. 2 shows a flowchart of the multi-robot arm cooperation method based on robot work experience accumulation in the embodiment of the specification. In the figure, the process of controlling each robot arm of a target robot is described. The order of steps listed in the embodiment is only one of the many step execution orders, and does not represent the only execution order. In actual system or device product execution, the method order shown in the embodiment or the drawing can be executed in sequence or in parallel.

[0057] Specifically, as shown in FIG. 2, the method can include:

[0058] Step 201: acquiring a task executed by a target robot and environment perception information of a working environment in which the target robot is located;

[0059] Step 202: determining an experience action of each execution moment of the target robot executing the task from a pre-established experience action database according to the task and the environment perception information;

[0060] Step 203: acquiring visual data, tactile data, mechanical data, joint state information and operation information of each robot arm of the target robot at a current execution moment;

[0061] Step 204: taking all the visual data, tactile data, mechanical data, joint state information and operation information of all the robot arms of the target robot at the current execution moment and the environment perception information as state variables corresponding to the current execution moment, and inputting the state variables into a pre-trained reinforcement learning model corresponding to the task for calculation to obtain a target action of a next execution moment of the target robot;

[0062] Step 205: comparing and evaluating the target action of the next execution moment and the experience action corresponding to the next execution moment to obtain a weighting factor;

[0063] Step 206: performing weighted calculation on the target action of the next execution moment and the experience action of the next execution moment by using the weighting factor to obtain an execution action of the next execution moment of the target robot;

[0064] Step 207: Control the work of each robotic arm of the target robot at the next execution time according to the execution action control at the next execution time.

[0065] With the embodiments of the present specification, first, the task executed by the target robot and the environment perception information of the working environment in which the target robot is located are acquired, and then the experience action corresponding to both the task and the environment perception information is acquired from the experience action database. When the state information of each robotic arm at the current execution time is acquired, the state information is taken as the state variable at the current execution time, and the target action at the next time corresponding to the state variable at the current time is calculated by using the reinforcement learning model. The target action is calculated according to the current state of each robotic arm and the task. Then, the target action and the experience action at the next execution time are weighted and calculated to obtain the optimal execution action, so as to control each robotic arm of the target robot by using the optimal execution action.

[0066] It should be noted that the weight distribution of the weighted calculation needs to be designed according to the actual application scene and target, so as to ensure that the system can effectively combine real-time information and historical experience to achieve optimal decision and control.

[0067] In the embodiments of the present specification, the length of time between two adjacent execution times of the target robot executing the task can be pre-set by the staff. The process of the target robot executing the task is divided into multiple execution times in the embodiments of the present specification. First, the experience action is acquired, which is the motion instruction and operation information of each robotic arm at each execution time when the same target robot executes the same task in the same working environment. Then, the target action at the next execution time is calculated according to the state variable at the current execution time by using the trained reinforcement learning model. Then, the target action at the next time and the experience action at the corresponding execution time are weighted and calculated to obtain the final execution action at the next time, and each robotic arm of the target robot is controlled.

[0068] In the embodiments of the present specification, the task executed by the target robot can include a grasping task, a moving task, a placing task, etc. The target robot can be an automated robot on a production line, which has multiple robotic arms, such as a multi-robotic arm system facing a building module connection installation process, an automobile assembly robot, etc., which are not limited by the embodiments of the present specification.

[0069] The environment perception information can be acquired by deploying various sensors in the working environment, including environment state data such as temperature, humidity, illumination, and other environmental parameters, and object positions, obstacle positions, etc. in the working environment.

[0070] The visual data includes image data collected by a visual sensor at the end of the robot arm, including RGB images, depth images, point cloud data, etc. The tactile data includes contact force and contact position collected by a tactile sensor at the end of the robot arm. The mechanical data includes joint torque and end force collected by a force sensor of the robot arm. The joint state information includes joint angle, velocity and acceleration of the robot arm. The operation information includes the operation type of the robot arm, such as grasping, rotating, placing and the like.

[0071] In the embodiments of the present application, the experience action database can store cases under different tasks and environmental perception information, including the experience action of each execution moment of each robot arm of the target robot when performing different tasks.

[0072] As shown in FIG. 3, all visual data, tactile data, mechanical data, joint state information, operation information and environmental perception information of the target robot at the current execution moment are further included as the state variable corresponding to the current execution moment.

[0073] Step 301: For each robot arm, the visual data, tactile data, mechanical data, joint state information, operation information and environmental perception information of the robot arm are fused to obtain the state vector of the robot arm.

[0074] Step 302: The state vectors of all robot arms form the state variable.

[0075] In the embodiments of the present application, an analog experimental environment can be constructed to train the data of the robot performing each task to obtain a trained reinforcement learning model.

[0076] Exemplarily, the state space of the reinforcement learning model can include the position and attitude of the robot arm (joint angle of each robot arm, position and attitude of the end effector, etc.), environmental state (object position in the working environment, obstacle position, etc.), perception data (real-time data from visual, tactile and mechanical sensors), etc. The action space can include joint action (rotation angle or movement distance of each joint, etc.), end action (movement path, speed, grasping or releasing of the end effector, etc.), cooperation action (cooperation strategy adjustment between multiple robot arms, such as task allocation, path planning, etc.).

[0077] The reward function of the reinforcement learning model can include task completion reward (high reward given after successfully completing the task), collision penalty (penalty given when the robot arm collides with obstacles or other robot arms), efficiency reward (higher reward for shorter task completion time), energy consumption penalty (higher penalty for higher energy consumption), path smoothing penalty (higher reward for smoother action path).

[0078] The termination conditions of the reinforcement learning model can include task completion (successfully completing the specified task), collision occurrence (serious collision of the robot arm, causing the task to be unable to continue), time overrun (task execution time exceeds the preset threshold), and robot arm failure (any robot arm fails and cannot continue to operate).

[0079] Exemplarily, the training process of the reinforcement learning model is as follows:

[0080] 1. Model initialization:

[0081] Neural network design: Select a neural network structure, such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0082] Parameter initialization: Randomly initialize the weight parameters of the neural network.

[0083] 2. Data collection and preprocessing:

[0084] Simulation environment construction: Build a simulation environment to simulate the collaborative work scenario of multiple robot arms.

[0085] Data collection: Collect a large number of state-action-reward-next state (s, a, r, s') samples in the simulation environment.

[0086] 3. Training process:

[0087] Experience replay: Store the collected data in an experience pool, randomly extract a small batch of data from it for training, and break the data correlation.

[0088] Target network: Introduce a target network, regularly update the parameters of the target network to stabilize the training process.

[0089] Policy update: According to the current state, select the optimal action (ε-greedy policy), and obtain the new state and reward by executing the action.

[0090] Gradient descent: Use the gradient descent algorithm (such as Adam) to update the neural network parameters to minimize the loss of the Q value or policy function. Q function

[0091] Periodic evaluation: Regularly evaluate the model performance and adjust the hyperparameters (learning rate, discount factor, etc.) to optimize the training effect.

[0092] 4. Model optimization:

[0093] Introduce enhanced strategies: such as double Q network, priority experience replay, A3C, etc. to improve the convergence speed and stability of the model.

[0094] Exploration and utilization balance: Adjust the ε value (exploration probability) to balance between exploring new strategies and utilizing existing strategies.

[0095] Multi-agent reinforcement learning: In the multi-robot collaborative work scenario, a multi-agent reinforcement learning method (such as MADDPG) is used to improve the overall collaboration efficiency.

[0096] 5. Model verification and testing:

[0097] Simulation environment verification: Test the trained model in a simulation environment and evaluate its performance in different tasks and environments.

[0098] Actual environment testing: Conduct experiments in a real multi-robot system to verify the practicality and robustness of the model.

[0099] It should be noted that the training process of the reinforcement learning model is well known in the art, and the embodiments of the present specification will not be repeated.

[0100] According to one embodiment of the present specification, as shown in FIG. 4, the state variable is input into the pre-trained reinforcement learning model corresponding to the task for calculation to obtain the target action of the target robot at the next execution time, which further comprises:

[0101] Step 401: Calculate the Q value between the state variable and all action variables in the action space of the reinforcement learning model;

[0102] Step 402: Take the action variable with the maximum Q value as the target action.

[0103] In the embodiments of the present specification, the target action, the experience action, and the execution action all include the motion instructions and operation information of each robot, wherein the motion instructions include joint actions, end actions, collaboration actions, etc., and the operation information includes grasping, moving, placing, etc.

[0104] It should be noted that the calculation process of the reinforcement learning model is well known in the art, and the embodiments of the present specification will not be repeated.

[0105] According to one embodiment of the present specification, the target action at the next execution time and the experience action corresponding to the next execution time are compared and evaluated to obtain a trade-off factor, which further comprises:

[0106] The action difference between the target action and the experience action is calculated by the formula Δα = |α target -α exp |, wherein Δα represents the action difference, α target represents the target action, and α exp represents the experience action;

[0107] The trade-off factor w is calculated by the formula w = 1 / (1+e α(Δα-β) ), wherein w represents the trade-off factor, and α and β represent adjustment parameters.

[0108] In the embodiments of the present specification, the adjustment parameters a and β can be empirical values, and the weighting factor w is a number between 0 and 1.

[0109] Further, the weighting calculation of the target action and the experience action by using the weighting factor to obtain the execution action of the target robot further includes:

[0110] The weighted target action is calculated by the formula , wherein represents the weighted target action.

[0111] The weighted experience action is calculated by the formula , wherein represents the weighted experience action.

[0112] The execution action is calculated by the formula , wherein a final represents the execution action.

[0113] In the embodiments of the present specification, if the experience action a exp and the target action have a difference of 0, it means that there is no difference between the experience action a exp and the target action , and they are equal. According to the execution action a final = (1-w)·a target + w·a exp , the execution action a final = a target = a exp .

[0114] Through the above steps and formulas, the robot behavior response strategy guided by experience knowledge can be realized, so that it realizes multi-robot parallel operation, autonomously learns the optimal control strategy, completes distributed or centralized coordination control of multi-robot arms, and thus improves the intelligence and generalization ability of the robot arm.

[0115] Based on the same inventive concept, the embodiments of the present specification also provide a multi-robot arm cooperation device based on accumulation of robot operation experience, as shown in FIG. 5, the device includes:

[0116] A task and environment information acquisition unit 501 is configured to acquire the task executed by the target robot and the environment perception information of the working environment thereof.

[0117] The experience action acquisition unit 502 is configured to determine, according to the task and the environment perception information, an experience action of each execution moment of the target robot performing the task from a pre-established experience action database.

[0118] The mechanical arm state acquisition unit 503 is configured to acquire visual data, tactile data, mechanical data, joint state information and operation information of each mechanical arm of the target robot at a current execution moment.

[0119] The target action calculation unit 504 is configured to take all of the visual data, tactile data, mechanical data, joint state information and operation information of each mechanical arm of the target robot at the current execution moment and the environment perception information as state variables corresponding to the current execution moment, and input the state variables into a pre-trained reinforcement learning model corresponding to the task to calculate a target action of a next execution moment of the target robot.

[0120] The trade-off factor calculation unit 505 is configured to compare and evaluate the target action of the next execution moment and an experience action corresponding to the next execution moment to obtain a trade-off factor.

[0121] The execution action calculation unit 506 is configured to perform weighted calculation on the target action of the next execution moment and the experience action of the next execution moment by using the trade-off factor to obtain an execution action of the next execution moment of the target robot.

[0122] The control unit 507 is configured to control each mechanical arm of the target robot to work at the next execution moment according to the execution action of the next execution moment.

[0123] Further, inputting the state variables into the pre-trained reinforcement learning model corresponding to the task to calculate the target action of the next execution moment of the target robot further includes:

[0124] calculating Q values between the state variables and all action variables in an action space of the reinforcement learning model;

[0125] taking an action variable with the largest Q value as the target action.

[0126] Further, comparing and evaluating the target action of the next execution moment and the experience action corresponding to the next execution moment to obtain the trade-off factor further includes:

[0127] calculating an action difference between the target action and the experience action by a formula Δα=|α target -α exp |, wherein Δα represents the action difference, α target represents the target action, and α exprepresenting an experience action;

[0128] The weighting factor is calculated by the formula w=1 / (1+e α(Δα-β) ), wherein w represents the weighting factor, and a and β represent adjustment parameters.

[0129] Further, the weighting factor is used to weight the target action and the experience action to obtain the execution action of the target robot, and the method further comprises:

[0130] The weighted target action is calculated by the formula , wherein represents the weighted target action.

[0131] The weighted experience action is calculated by the formula , wherein represents the weighted experience action.

[0132] The execution action is calculated by the formula , wherein a final represents the execution action.

[0133] Further, the visual data comprises image data collected by a visual sensor at the end of the robot arm, the tactile data comprises contact force and contact position collected by a tactile sensor at the end of the robot arm, the mechanical data comprises joint torque and end force collected by a force sensor of the robot arm, the joint state information comprises joint angle, velocity and acceleration of the robot arm, and the operation information comprises operation type of the robot arm.

[0134] Further, all of the visual data, tactile data, mechanical data, joint state information, operation information and environment perception information of the target robot at the current execution time are taken as state variables corresponding to the current execution time, and the method further comprises:

[0135] For each robot arm, the visual data, tactile data, mechanical data, joint state information, operation information and environment perception information of the robot arm are fused to obtain a state vector of the robot arm.

[0136] The state vectors of all robot arms form the state variables.

[0137] Further, the execution action comprises motion instructions and operation information of each robot arm.

[0138] Since the principle of solving the problem of the above device is similar to the above method, the implementation of the above system can refer to the implementation of the above method, and the repeated parts will not be described again.

[0139] As shown in FIG. 6, a computer device of an embodiment of the present specification is shown in a structural diagram. The computer device of the embodiment of the present specification can run the feature processing platform of the embodiment of the present specification.

[0140] The computer device 602 can include one or more processors 604, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 602 can also include any memory 606 for storing any kind of information, such as code, settings, data, etc. Without limitation, for example, the memory 606 can include any one or combination of: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, etc. More generally, any storage resource can store information using any technology.

[0141] Further, any storage resource can provide volatile or non-volatile retention of information.

[0142] Further, any storage resource can represent a fixed or removable component of the computer device 602. In one case, the computer device 602 can perform any operation of the associated instructions when the processor 604 executes the associated instructions stored in any storage resource or combination of storage resources. The computer device 602 also includes one or more drive mechanisms 608, such as a hard disk drive system, an optical disk drive system, etc., for interacting with any storage resource.

[0143] The computer device 602 can also include an input / output module 610 (I / O) for receiving various inputs (via input devices 612) and for providing various outputs (via output devices 614). One particular output mechanism can include a presentation device 616 and an associated graphical user interface (GUI) 618. In other embodiments, the input / output module 610 (I / O), the input devices 612, and the output devices 614 can also not be included, just as a computer device in a network. The computer device 602 can also include one or more network interfaces 620 for exchanging data with other devices via one or more communication links 622. One or more communication buses 624 couple the above-described components together.

[0144] The communication links 622 can be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication links 622 can include any combination of hardwired links, wireless links, routers, gateway functionality, name servers, etc., governed by any protocol or combination of protocols.

[0145] It should be noted that when the Kubernetes platform of the embodiments of the present specification implements the method described in the embodiments of the present specification for the computer device 502 described in the embodiments of the present specification, the presentation device 516 and the associated graphical user interface (GUI) 518 and the like can also not be included. For example, only a computer minimum system consisting of a processing device 504, a storage resource 506 and a network interface 520.

[0146] The embodiments of the present specification also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0147] The embodiments of the present specification also provide a computer readable instruction, wherein when the processor executes the instruction, the program in the instruction causes the processor to execute the above method.

[0148] It should be understood that in various embodiments of the present application, the size of the sequence number of each process described above does not mean the order of execution, the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

Claims

1. A multi-robot arm collaboration method based on accumulation of robot work experience, characterized by, The method comprises: acquiring a task executed by a target robot and environment perception information of a working environment in which the target robot is located; determining, according to the task and the environment perception information, an experience action of each execution time of the target robot executing the task from a pre-established experience action database; acquiring visual data, tactile data, mechanical data, joint state information and operation information of each robot arm of the target robot at a current execution time; inputting all of the visual data, tactile data, mechanical data, joint state information, operation information of all robot arms of the target robot at the current execution time and the environment perception information as state variables corresponding to the current execution time into a pre-trained reinforcement learning model corresponding to the task for calculation to obtain a target action of the target robot at a next execution time; comparing and evaluating the target action at the next execution time and an experience action corresponding to the next execution time to obtain a weighting factor; performing weighted calculation on the target action at the next execution time and the experience action at the next execution time by using the weighting factor to obtain an execution action of the target robot at the next execution time; controlling each robot arm of the target robot to work at the next execution time according to the execution action at the next execution time.

2. The method of claim 1, wherein, The inputting the state variables into the pre-trained reinforcement learning model corresponding to the task for calculation to obtain the target action of the target robot at the next execution time further comprises: calculating Q values between the state variables and all action variables in an action space of the reinforcement learning model; taking an action variable with the largest Q value as the target action.

3. The method of claim 1, wherein, The comparing and evaluating the target action at the next execution time and the experience action corresponding to the next execution time to obtain the weighting factor further comprises: By the formula Δa = |a target -a exp | the action difference between the target action and the experience action is calculated, where Δa represents the action difference, a target represents the target action, a exp represents the experience action; The weighting factor is calculated by the equation w = 1 / (1+e α(Δa-β) ), where w represents the weighting factor, and a and β represent the adjustment parameters.

4. The method of claim 3, wherein, performing weighted calculation on the target action and the experience action by using the weighting factor to obtain the execution action of the target robot further comprises: By the formula computing a weighted target action, wherein representing the weighted target action; By the formula computing a weighted experience action, wherein representing the weighted experience action. By the formula computing the performance action, wherein a final representing a performance action.

5. The method of claim 1, wherein, The visual data comprises image data collected by a visual sensor at the end of the robot arm, the tactile data comprises contact force and contact position collected by a tactile sensor at the end of the robot arm, the mechanical data comprises joint torque and end force collected by a force sensor of the robot arm, the joint state information comprises joint angle, speed and acceleration of the robot arm, and the operation information comprises operation type of the robot arm.

6. The method of claim 1, wherein, The inputting all of the visual data, tactile data, mechanical data, joint state information, operation information of all robot arms of the target robot at the current execution time and the environment perception information as state variables corresponding to the current execution time further comprises: for each robot arm, fusing the visual data, tactile data, mechanical data, joint state information, operation information and environment perception information of the robot arm to obtain a state vector of the robot arm; forming the state vectors of all robot arms into the state variables.

7. The method of claim 1, wherein, The execution action comprises motion instructions and operation information of each robot arm.

8. A multi-arm collaborative device based on robot job experience accumulation, characterized by, The device comprises: A task and environment information acquisition unit is configured to acquire a task performed by a target robot and environment perception information of a working environment in which the target robot is located. An experience action acquisition unit is configured to determine, according to the task and the environment perception information, an experience action of each execution moment of the target robot performing the task from a pre-established experience action database. A robot arm state acquisition unit is configured to acquire visual data, tactile data, mechanical data, joint state information and operation information of each robot arm of the target robot at a current execution moment. A target action calculation unit is configured to take all of the visual data, tactile data, mechanical data, joint state information and operation information of the robot arm of the target robot at the current execution moment and the environment perception information as state variables corresponding to the current execution moment, and input the state variables into a pre-trained reinforcement learning model corresponding to the task to perform calculation, so as to obtain a target action of a next execution moment of the target robot. A trade-off factor calculation unit is configured to compare and evaluate the target action of the next execution moment and an experience action corresponding to the next execution moment, so as to obtain a trade-off factor. An execution action calculation unit is configured to perform weighted calculation on the target action of the next execution moment and the experience action of the next execution moment by using the trade-off factor, so as to obtain an execution action of the next execution moment of the target robot. A control unit is configured to control each robot arm of the target robot to work at the next execution moment according to the execution action of the next execution moment. The processor executes the computer program to implement the method in any one of claims 1 to 7.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein, The computer readable storage medium stores the computer program, and the computer program is executed by the processor to implement the method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, ​

Images