A dual-arm robot skill learning method based on Bayesian interaction primitives

Abstract

The invention discloses a dual-arm robot skill learning method based on Bayesian interaction primitives, builds a dual-arm robot skill learning platform to control the dual-arm robot and collect data; teaches the dual-arm robot through remote operation and collects expert teaching data; establishes a Bayesian interaction primitive learning framework and uses basis functions to fit the teaching trajectory; obtains multimodal data of the current state through multiple sensors, uses sequence alignment and dynamic capture methods to perform spatiotemporal reasoning to obtain the phase of the current state, and the Bayesian interaction primitive outputs the posterior distribution of the future action sequence of the dual-arm robot according to the current phase; each time step is weighted averaged according to the corresponding actions of the current and historical output action sequences and then executed until the phase estimation task is completed. The skill learning efficiency is high, the teaching method is simple and easy, and the skill learning effect is excellent, which effectively solves the problems of slow skill development of dual-arm robots, poor coordination of dual-arm operations, and low operation fluency.

Description

Technical Field

[0001] The invention belongs to the technical field of artificial intelligence, and in particular relates to a dual-arm robot skill learning method based on Bayesian interaction primitives. Background Art

[0002] With the development of the intelligent manufacturing industry and the promotion of embodied intelligent robots, traditional single-arm collaborative robots and industrial robots in many scenarios of the manufacturing and service industries are gradually being replaced by dual-arm robots. For example, dual-arm robots are used in the manufacturing industry to complete collaborative handling, aircraft long-beam assembly, etc., and in smart home service scenarios, such as folding clothes, cleaning, and washing dishes. The dual-arm robot has two independently controlled mechanical arms, which makes its operation more flexible and adaptable. The dual-arm collaboration significantly improves its efficiency and accuracy when performing tasks. Therefore, compared with single-arm robots, dual-arm robots can more efficiently handle tasks with larger loads and more complex scenarios by virtue of their own advantages.

[0003] However, some recent studies have shown that the acquisition methods of dual-arm robots' operating skills need to be improved. At present, the robot's skill learning methods mainly include traditional methods that rely on precise control, sensor feedback and pre-programming, and machine learning methods through learning and training. Some scholars have proposed the use of visual servoing methods, using cameras and image processing technology to guide the robot's movement, using force sensors to sense and adjust the robot's operating force, and using pre-programming to plan the path and action sequence of the dual-arm robot to perform tasks in complex task scenarios. Although these methods are highly robust, they require a lot of time and energy to adjust the impedance coefficient, pre-programming, etc., and have no scene generalization ability, and can only perform set tasks and operation objects. Recently, some scholars have proposed machine learning-based methods to enable dual-arm robots to acquire operating skills, such as using imitation learning behavior to clone human operating skills. Although imitation learning can efficiently learn human operating experience and complete operating tasks with both arms in a collaborative manner, the disadvantage of this method is that when the dual-arm robot is disturbed by external factors or other reasons and is in a state it has never seen before, it cannot make decisions like humans to continue to complete the task. Reinforcement learning is a method of learning the best strategy through trial and error. In the process of interacting with the environment, the dual-arm robot obtains the best operation strategy by maximizing rewards. Although this method can eventually obtain the best operation strategy, the reinforcement learning training time is often very long, and the coordination of the two arms is not high. At the same time, during the training process, the two arms are prone to collision and damage to the equipment, the reward function is difficult to shape, and the loss function is not easy to converge.

[0004] Based on this, the present invention proposes a dual-arm robot skill learning method based on Bayesian interaction primitives. Summary of the invention

[0005] In view of the above technical problems, the present invention provides a dual-arm robot skill learning method based on Bayesian interaction primitives.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A dual-arm robot skill learning method based on Bayesian interaction primitives, the method comprising the following steps:

[0008] S100: Build a dual-arm robot skill learning platform to control the dual-arm robot and collect data in real time and efficiently;

[0009] S200: Teach the dual-arm robot through remote control and collect expert teaching data;

[0010] S300: Establish a Bayesian interactive primitive learning framework and use basis functions to fit the teaching trajectory to encode and learn the operation skills of human experts;

[0011] S400: The multimodal data of the current state is obtained through multiple sensors, and the sequence alignment and dynamic capture methods are used for spatiotemporal reasoning to obtain the phase of the current state. The Bayesian interaction primitive outputs the posterior distribution of the future action sequence of the dual-arm robot according to the current phase;

[0012] S500: Each time step is executed after weighted averaging the corresponding actions of the current and historical output action sequences. In the perception-prediction-execution cycle, the system continues to run until the phase estimation task is completed.

[0013] Preferably, S100 includes:

[0014] S110: Install the two-finger gripper at the end of the robot and debug the dual-arm robot teaching pendant to make the hardware system meet the various task operations taught by experts;

[0015] S120: Build the communication architecture between the dual-arm human controller and the gripper controller, collect the teaching data of the dual-arm robot through the relevant interface, and control its dual arms and grippers at the same time;

[0016] S130: Write robot control code, instantiate left and right robotic arms and left and right end grippers, and use multi-threading to ensure that the dual-arm robot operating system can run in real-time and efficiently.

[0017] Preferably, S120 includes:

[0018] Use Socket and TCP communication to realize the host controlling two robotic arms. First, make sure that the host and the controllers of the left and right robotic arms are in the same LAN, and assign fixed IP addresses and port numbers to the host and the two robotic arm controllers; connect the left and right gripper controllers through serial communication, and set RS485 communication parameters;

[0019] The interfaces corresponding to the robot arm and the gripper are called to collect the joint angles, joint torques, end position and posture of the robot arm, and the opening and closing status of the gripper, and the robot arm and the gripper are controlled through the interfaces corresponding to the robot arm and the gripper.

[0020] Preferably, S200 includes:

[0021] S210: The operator teaches the dual-arm robot through two teaching pendants, completes the preset operation task, and collects the teaching data during the teaching process. The teaching data includes the joint angle position, joint speed, joint torque, end position and gripper opening and closing state of the dual-arm robot;

[0022] S220: Determine the transformation matrix of the right arm base coordinate system relative to the left arm base coordinate system, and transform the end position and posture teaching data of the right arm of the dual-arm robot from the base coordinate system of the right arm to the base coordinate system of the left arm.

[0023] Preferably, S220 is specifically:

[0024] S221: Measuring translation vector , the translation vector Describes the position of the origin of the right arm base coordinate system relative to the origin of the left arm base coordinate system; assuming that the position of the origin of the right arm base coordinate system in the left arm base coordinate system is ,but:

[0025] ;

[0026] S222: Measure the rotation matrix of the right arm base coordinate system relative to the left arm base coordinate system , if the rotation angle around the X axis is known , rotation angle around the Y axis , rotation angle around the Z axis , then the rotation matrix It can be expressed as:

[0027] ;

[0028] in They are the rotation matrices around the X, Y, and Z axes respectively;

[0029] If the quaternion is known , then the rotation matrix It can be calculated by the following formula:

[0030] ;

[0031] S223: Constructing transformation matrix The posture of the right arm end in the right arm base coordinate system is transformed to the left arm base coordinate system. The transformation matrix obtained from S221 and S222 is as follows:

[0032] ;

[0033] At this time, it is assumed that the pose matrix of the right arm end in the right arm base coordinate system is , then the right arm end is converted to the pose matrix of the left arm base coordinate system for:

[0034] .

[0035] Preferably, S300 includes:

[0036] S310: Define a time-varying sequence Y:

[0037] ;

[0038] Where T is the observation step length of the teaching data, and D is the observation dimension of the teaching data;

[0039] S320: Each state dimension of sequence Y are approximated using weighted linear combinations of time-dependent nonlinear basis functions:

[0040] ;

[0041] in, The first Observation dimensions, Represents time, basis function , corresponding to the weight of the basis function , relative phase value Replace absolute time ,and , represents the approximation error, the basis functions include nonlinear functions with time dependence;

[0042] S330: Obtain the weight of the basis function by the least squares method , the weights of each dimension are combined to obtain a Bayesian interaction primitive model of the dual-arm robot's skills:

[0043] ;

[0044] in, , the weights can be solved by linear regression method;

[0045] S340: Extract the probability distribution of weight vectors from all given teachings , sampling from this distribution can generate sample trajectories containing all observed sensor states and controls in the dual-arm robot manipulation task, so the definition of a given interaction of Observation , based on the robot's previous state, we get the posterior distribution about the future state and control:

[0046] .

[0047] Preferably, S400 includes:

[0048] S410: According to the currently sensed multimodal data, phase estimation is performed using sequence alignment and dynamic capture methods to obtain the completion progress of the current operation task of the dual-arm robot;

[0049] S420: The Bayesian interaction primitive outputs the posterior distribution of the future action sequence according to the phase result of S410, and then updates the covariance and Kalman gain.

[0050] Preferably, S410 specifically includes:

[0051] The given length is Average teaching trajectory and the trajectory where only T steps are observed , by dynamically capturing the optimal alignment, that is, minimizing the distance function :

[0052] ;

[0053] ;

[0054] ;

[0055] in, From the trajectory From the beginning to the The cumulative distance of the steps, is from the beginning of the average teaching trajectory to the Cumulative distance of steps, calculated recursively By comparing the minimum value of the three cases plus the current distance To update , ,or Path selection; recursion in the calculation to reach The minimum cumulative distance is terminated, where , and then get the estimated phase , indicating the trajectory In the average trajectory The position in Represents the index in the cumulative distance calculation process, which is used to traverse and accumulate the distance from the starting point to the current point.

[0056] Preferably, S420 includes:

[0057] S421: Based on S340, use vector augmentation to construct a new state vector , the state vector includes the weights of the basis functions , Phase and phase velocity , so the posterior distribution is:

[0058] ;

[0059] Among them, the current status The posterior distribution of It is determined by two factors: the current state In the case of Probability And given all previous t-1 step observations and the initial state In the case of the current state Prior distribution of predictions;

[0060] S422: Define a A collection of members , where the number of sets is not less than the number of teaching samples, that is, , approximated by propagating each collection element forward one time step , the state propagation formula is:

[0061] ;

[0062] In the formula, represents the predicted state vector of the jth member at time step t, represents the updated state vector of the jth member at time step t, is a state transition operator with constant velocity, The noise error is used to simulate possible random disturbances or uncertainties in the real system;

[0063] S423: Based on the observation results and nonlinear observation operator Update collection members:

[0064] ;

[0065] ;

[0066] For each member of the set Through the nonlinear observation operator Calculate predicted observed values , and then calculate the observation deviation matrix , which contains the deviation of each member from the mean, where j=1,2,…,E;

[0067] S424: Calculate innovation covariance:

[0068] ;

[0069] In the formula, represents the innovation at time step t, is the deviation of the observation noise matrix;

[0070] S425: Calculate Kalman gain:

[0071] ;

[0072] ;

[0073] In the formula, is the deviation matrix of the set members, Represents the transpose of the bias matrix in observation space.

[0074] Preferably, S500 includes:

[0075] Assume that at time step t, the action sequence of the next 3 steps output by the Bayesian interaction primitive is , and at this time t>2, and the historical action sequences output at t-1 and t-2 are and , then the current execution action is executed by weighted average of the corresponding actions of the current output action sequence and the historical output action sequence, that is:

[0076] ;

[0077] If t<2, then ,After waiting for the robot to complete the action, it continues to execute S400 and S500. In the perception-prediction-execution cycle, the system continues to run until the phase estimation task is completed.

[0078] The above-mentioned dual-arm robot skill learning method based on Bayesian interaction primitives designs a framework that integrates expert teaching and Bayesian interaction primitives for dual-arm robot skill learning, which improves the efficiency, collaborative operation ability, operation fluency and robustness of dual-arm robot skill learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a flow chart of a dual-arm robot skill learning method based on Bayesian interaction primitives in one embodiment of the present invention;

[0080] Figure 2 It is an architecture diagram of a dual-arm robot skill learning platform of a dual-arm robot skill learning method based on Bayesian interaction primitives in one embodiment of the present invention;

[0081] Figure 3 The present invention is a detailed test operation flow chart of a dual-arm robot skill learning method based on Bayesian interaction primitives in one embodiment of the present invention. DETAILED DESCRIPTION

[0082] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0083] In one embodiment, Figure 1 As shown, a dual-arm robot skill learning method based on Bayesian interaction primitives, the method comprising the following steps:

[0084] S100: Build a dual-arm robot skill learning platform to control the dual-arm robot and collect data in real time and efficiently;

[0085] S200: Teach the dual-arm robot through remote control and collect expert teaching data;

[0086] S300: Establish a Bayesian interactive primitive learning framework and use basis functions to fit the teaching trajectory to encode and learn the operation skills of human experts;

[0087] S400: Obtain multimodal data of the current state through multiple sensors (vision, position, force, etc.), use sequence alignment and dynamic capture methods to perform spatiotemporal reasoning to obtain the phase of the current state, and the Bayesian interaction primitive outputs the posterior distribution of the future action sequence of the dual-arm robot based on the current phase;

[0088] S500: Each time step is executed after weighted averaging the corresponding actions of the current and historical output action sequences. In the perception-prediction-execution cycle, the system continues to run until the phase estimation task is completed.

[0089] The above-mentioned dual-arm robot skill learning method based on Bayesian interaction primitives designs a framework that integrates expert teaching and Bayesian interaction primitives for dual-arm robot skill learning, which improves the efficiency, collaborative operation ability, operation fluency and robustness of dual-arm robot skill learning.

[0090] In one embodiment, Figure 2 As shown, S100 includes:

[0091] S110: Install the two-finger gripper at the end of the robot and debug the dual-arm robot teaching pendant to make the hardware system meet the various task operations taught by experts;

[0092] S120: Build the communication architecture between the dual-arm human controller and the gripper controller, collect the teaching data of the dual-arm robot through the relevant interface, and control its dual arms and grippers at the same time;

[0093] S130: Write robot control code, instantiate left and right robotic arms and left and right end grippers, and use multi-threading to ensure that the dual-arm robot operating system can run in real-time and efficiently.

[0094] In one embodiment, S110 includes:

[0095] S110: Prepare a dual-arm robot, two teaching pendants and two grippers, and set the teaching mode to Cartesian space operation mode, so that the teaching operator can intuitively control the robot arm to move along the desired path;

[0096] S120: Install the gripper at the end of the robot and control the gripper via RS485 serial communication.

[0097] In one embodiment, S120 includes:

[0098] Use Socket and TCP communication to realize the host controlling two robotic arms. First, make sure that the host and the controllers of the left and right robotic arms are in the same LAN, and assign fixed IP addresses and port numbers to the host and the two robotic arm controllers; connect the left and right gripper controllers through serial communication, and set RS485 communication parameters;

[0099] The interfaces corresponding to the robot arm and the gripper are called to collect the joint angles, joint torques, end position and posture of the robot arm, and the opening and closing status of the gripper, and the robot arm and the gripper are controlled through the interfaces corresponding to the robot arm and the gripper.

[0100] In one embodiment, S200 includes:

[0101] S210: The operator teaches the dual-arm robot through two teaching pendants, completes the preset operation task, and collects the teaching data during the teaching process. The teaching data includes the joint angle position, joint speed, joint torque, end position and gripper opening and closing state of the dual-arm robot;

[0102] S220: Determine the transformation matrix of the right arm base coordinate system relative to the left arm base coordinate system, and transform the end position and posture teaching data of the right arm of the dual-arm robot from the base coordinate system of the right arm to the base coordinate system of the left arm.

[0103] In one embodiment, S210 specifically includes:

[0104] The teaching personnel remotely control the dual-arm robot through the teaching pendant to complete preset operation tasks, such as collaborative handling, pouring water, etc. The same task requires the collection of about 20 to 30 sets of teaching data, and the collected teaching data are pre-processed to discard the teaching data whose quality does not meet the preset requirements. The teaching data includes the joint angular position, joint angular velocity, joint torque, and end-position gripper opening and closing status of the dual-arm robot.

[0105] In one embodiment, S220 is specifically:

[0106] S221: Measuring translation vector , the translation vector Describes the position of the origin of the right arm base coordinate system relative to the origin of the left arm base coordinate system; assuming that the position of the origin of the right arm base coordinate system in the left arm base coordinate system is ,but:

[0107] ;

[0108] S222: Measure the rotation matrix of the right arm base coordinate system relative to the left arm base coordinate system , if the rotation angle around the X axis is known , rotation angle around the Y axis , rotation angle around the Z axis , then the rotation matrix It can be expressed as:

[0109] ;

[0110] in They are the rotation matrices around the X, Y, and Z axes respectively;

[0111] If the quaternion is known , then the rotation matrix It can be calculated by the following formula:

[0112] ;

[0113] S223: Constructing transformation matrix The posture of the right arm end in the right arm base coordinate system is transformed to the left arm base coordinate system. The transformation matrix obtained from S221 and S222 is as follows:

[0114] ;

[0115] At this time, it is assumed that the pose matrix of the right arm end in the right arm base coordinate system is , then the right arm end is converted to the pose matrix of the left arm base coordinate system for:

[0116] .

[0117] In one embodiment, S300 includes:

[0118] S310: Define a time-varying sequence Y:

[0119] ;

[0120] Where T is the observation step length of the teaching data, and D is the observation dimension of the teaching data;

[0121] S320: Each state dimension of sequence Y are approximated using weighted linear combinations of time-dependent nonlinear basis functions:

[0122] ;

[0123] in, The first Observation dimensions, Represents time, basis function , corresponding to the weight of the basis function , relative phase value Replace absolute time ,and , Represents the approximation error, and the basis functions include nonlinear functions including time dependence such as Gaussian function and Sigmod function;

[0124] S330: Obtain the weight of the basis function by the least squares method , the weights of each dimension are combined to obtain a Bayesian interaction primitive model of the dual-arm robot's skills:

[0125] ;

[0126] in, , the weights can be solved by linear regression method;

[0127] S340: Extract the probability distribution of weight vectors from all given teachings , sampling from this distribution can generate sample trajectories containing all observed sensor states and controls in the dual-arm robot manipulation task, so the definition of a given interaction of Observation , based on the robot's previous state, we get the posterior distribution about the future state and control:

[0128] .

[0129] In one embodiment, Figure 3 As shown, S400 includes:

[0130] S410: According to the currently sensed multimodal data, phase estimation is performed using sequence alignment and dynamic capture methods to obtain the completion progress of the current operation task of the dual-arm robot;

[0131] S420: The Bayesian interaction primitive outputs the posterior distribution of the future action sequence according to the phase result of S410, and then updates the covariance and Kalman gain.

[0132] In one embodiment, S410 specifically includes:

[0133] The given length is Average teaching trajectory and the trajectory where only T steps are observed , by dynamically capturing the optimal alignment, that is, minimizing the distance function :

[0134] ;

[0135] ;

[0136] ;

[0137] in, From the trajectory From the beginning to the The cumulative distance of the steps, is from the beginning of the average teaching trajectory to the Cumulative distance of steps, calculated recursively By comparing the minimum value of the three cases plus the current distance To update , ,or Path selection; recursion in the calculation to reach The minimum cumulative distance is terminated, where , and then get the estimated phase , indicating the trajectory In the average trajectory The position in Represents the index in the cumulative distance calculation process, which is used to traverse and accumulate the distance from the starting point to the current point.

[0138] In one embodiment, S420 includes:

[0139] S421: Based on S340, use vector augmentation to construct a new state vector , the state vector includes the weights of the basis functions , Phase and phase velocity , so the posterior distribution is:

[0140] ;

[0141] Among them, the current status The posterior distribution of It is determined by two factors: the current state In the case of Probability And given all previous t-1 step observations and the initial state In the case of the current state Prior distribution of predictions;

[0142] S422: Define a A collection of members , where the number of sets is not less than the number of teaching samples, that is, , approximated by propagating each collection element forward one time step , the state propagation formula is:

[0143] ;

[0144] In the formula, represents the predicted state vector of the jth member at time step t, represents the updated state vector of the jth member at time step t, is a state transition operator with constant velocity, The noise error is used to simulate possible random disturbances or uncertainties in the real system;

[0145] S423: Based on the observation results and nonlinear observation operator Update collection members:

[0146] ;

[0147] ;

[0148] For each member of the set Through the nonlinear observation operator Calculate predicted observed values , and then calculate the observation deviation matrix , which contains the deviation of each member from the mean, where j=1,2,…,E;

[0149] S424: Calculate innovation covariance:

[0150] ;

[0151] In the formula, represents the innovation at time step t, is the deviation of the observation noise matrix;

[0152] S425: Calculate Kalman gain:

[0153] ;

[0154] ;

[0155] In the formula, is the deviation matrix of the set members, Represents the transpose of the bias matrix in observation space.

[0156] In one embodiment, S500 includes:

[0157] Assume that at time step t, the action sequence of the next 3 steps output by the Bayesian interaction primitive is , and at this time t>2, and the historical action sequences output at t-1 and t-2 are and , then the current execution action is executed by weighted average of the corresponding actions of the current output action sequence and the historical output action sequence, that is:

[0158] ;

[0159] If t<2, then ,After waiting for the robot to complete the action, it continues to execute S400 and S500. In the perception-prediction-execution cycle, the system continues to run until the phase estimation task is completed.

[0160] Compared with the prior art, the advantages of the present invention are as follows: (1) A skill learning framework based on a dual-arm robot system is proposed, which can more conveniently collect data of the dual-arm robot and can also control the dual-arm robot in real time and efficiently. (2) Based on the idea of ​​imitation learning, the Bayesian interaction primitive is used to learn the teaching data of human experts, which greatly improves the skill learning efficiency and the collaborative operation ability of the dual arms. (3) In view of the problem that the original imitation learning method cannot handle the unseen state scene and has poor anti-interference ability, the present invention adopts the method of sequence alignment and dynamic capture to perform spatiotemporal estimation, obtains the current phase to replace the absolute time, avoids the dual-arm robot from entering the unseen state after being disturbed and cannot continue to correctly perform the operation task, and improves the robustness of the dual-arm robot in performing the operation task. (4) The present invention adopts the idea of ​​action sequence. The Bayesian interaction primitive outputs the future action sequence. The action of the current time step is determined by the weighted average of the historical prediction and the current prediction. Each time step performs perception-prediction-execution, which improves the smoothness of discrete switching between steps.

[0161] The present invention discloses a dual-arm robot skill learning method based on Bayesian interaction primitives. For the dual-arm robot skill learning, a framework integrating expert teaching and Bayesian interaction primitives is designed to improve the efficiency, collaborative operation capability, operation fluency and robustness of the dual-arm robot skill learning. The method first builds a dual-arm robot skill learning platform to ensure that the dual-arm robot can be controlled and data collected in real time and efficiently, and conveniently collects the teaching data such as the joint angle position, speed, torque, end position, and gripper state of the dual-arm robot through Socket, TCP and RS485 communication architecture. At the same time, the dual arms and grippers are controlled in a multi-threaded manner to ensure that the dual-arm robot operating system can operate in real-time and efficiently. The dual-arm robot is taught to perform operations such as pouring water and collaborative handling through teleoperation, and expert teaching data such as joint angle position, velocity, torque, end position and gripper opening and closing state are collected, and the end position data of the right arm is converted from the base coordinate system of the right arm to the base coordinate system of the left arm. Then, a Bayesian interaction primitive learning framework is built, and the teaching trajectory is fitted using a nonlinear basis function containing time dependencies. The teaching data is encoded to obtain the Bayesian interaction primitives to generate Bayesian interaction primitives and learn the operation skills of human experts. The Bayesian interaction primitive estimates the phase according to the current state through spatiotemporal reasoning. Subsequently, the Bayesian interaction primitive outputs the posterior distribution of the future action sequence according to the current phase, and each time step is executed after weighted averaging the corresponding actions of the current and historical output action sequences. In the perception-prediction-execution cycle, the system continues to run until the phase estimation task is completed.

[0162] The above is a detailed introduction to a dual-arm robot skill learning method based on Bayesian interaction primitives provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A dual-arm robot skill learning method based on Bayesian interaction primitives, characterized in that: The method comprises the following steps: S100: Build a dual-arm robot skill learning platform to control the dual-arm robot and collect data in real time and efficiently; S200: Teach the dual-arm robot through teleoperation and collect expert teaching data; S300: Establish a Bayesian interactive primitive learning framework and use basis functions to fit the teaching trajectory to encode and learn the operation skills of human experts; S300 includes: S310: Define time steps The changing sequence Y: ； Where T is the total number of observation steps of the teaching data, and D is the total number of observation dimensions of the teaching data; S320: Each state dimension of sequence Y are approximated using weighted linear combinations of time-dependent nonlinear basis functions: ； in, The first Observation dimensions, represents the tth time step among T time steps, represents the nonlinear observation operator, basis function , corresponding to the weight of the basis function , relative phase value Alternative time steps ,and , represents the approximation error, the basis functions include nonlinear functions with time dependence; S330: Obtain the weight of the basis function by the least squares method , the weights of each dimension are combined to obtain a Bayesian interaction primitive model of the dual-arm robot's skills: ； in, , the weights can be solved by linear regression method; S340: Extract the probability distribution of weight vectors from all given teachings , sampling from this distribution can generate sample trajectories containing all observed sensor states and controls in the dual-arm robot manipulation task, so the given sequence is defined as 1st to 2nd time step observation sequence , based on the robot's previous state, we get the posterior distribution about the future state and control: ； S400: The multimodal data of the current state is obtained through multiple sensors, and the sequence alignment and dynamic capture methods are used for spatiotemporal reasoning to obtain the phase of the current state. The Bayesian interaction primitive outputs the posterior distribution of the future action sequence of the dual-arm robot according to the current phase; S500: Each time step is executed after weighted averaging the corresponding actions of the current and historical output action sequences. In the perception-prediction-execution cycle, the system continues to run until the phase estimation task is completed.

2. The method according to claim 1, characterized in that S100 includes: S110: Install the two-finger gripper at the end of the robot and debug the dual-arm robot teaching pendant to make the hardware system meet the various task operations taught by experts; S120: Build the communication architecture between the dual-arm human controller and the gripper controller, collect the teaching data of the dual-arm robot through the relevant interface, and control its dual arms and grippers at the same time; S130: Write robot control code, instantiate left and right robotic arms and left and right end grippers, and use multi-threading to ensure that the dual-arm robot operating system can run in real-time and efficiently.

3. The method according to claim 2, characterized in that S120 includes: Use Socket and TCP communication to realize the host controlling two robotic arms. First, make sure that the host and the controllers of the left and right robotic arms are in the same LAN, and assign fixed IP addresses and port numbers to the host and the two robotic arm controllers; connect the left and right gripper controllers through serial communication, and set RS485 communication parameters; The interfaces corresponding to the robot arm and the gripper are called to collect the joint angles, joint torques, end position and posture of the robot arm, and the opening and closing status of the gripper, and the robot arm and the gripper are controlled through the interfaces corresponding to the robot arm and the gripper.

4. The method according to claim 1, characterized in that: S200 includes: S210: The operator teaches the dual-arm robot through two teaching pendants, completes the preset operation task, and collects the teaching data during the teaching process. The teaching data includes the joint angle position, joint speed, joint torque, end position and gripper opening and closing state of the dual-arm robot; S220: Determine the transformation matrix of the right arm base coordinate system relative to the left arm base coordinate system, and transform the end position and posture teaching data of the right arm of the dual-arm robot from the base coordinate system of the right arm to the base coordinate system of the left arm.

5. The method according to claim 4, characterized in that S220 is specifically: S221: Measuring translation vector , the translation vector Describes the position of the origin of the right arm base coordinate system relative to the origin of the left arm base coordinate system; assuming that the position of the origin of the right arm base coordinate system in the left arm base coordinate system is ,but: ； S222: Measure the rotation matrix of the right arm base coordinate system relative to the left arm base coordinate system , if the rotation angle around the X axis is known , rotation angle around the Y axis , rotation angle around the Z axis , then the rotation matrix It can be expressed as: ； in They are the rotation matrices around the X, Y, and Z axes respectively; If the quaternion is known , then the rotation matrix It can be calculated by the following formula: ； S223: Constructing transformation matrix The pose of the right arm end in the right arm base coordinate system is transformed to the left arm base coordinate system. The transformation matrix obtained from S221 and S222 is as follows: ； At this time, it is assumed that the pose matrix of the right arm end in the right arm base coordinate system is , then the right arm end is converted to the pose matrix of the left arm base coordinate system for: 。 6. The method according to claim 5, characterized in that S400 includes: S410: According to the currently sensed multimodal data, phase estimation is performed using sequence alignment and dynamic capture methods to obtain the completion progress of the current operation task of the dual-arm robot; S420: The Bayesian interaction primitive outputs the posterior distribution of the future action sequence according to the phase result of S410, and then updates the covariance and Kalman gain.

7. The method according to claim 6, characterized in that S410 is specifically: The given length is Average teaching sequence and only the sequence with a total of T time steps is observed , by dynamically capturing the optimal alignment, that is, minimizing the distance function : ； ； ； in, From the sequence From the beginning to the The cumulative distance of time steps, is the average teaching sequence from the beginning to the Cumulative distance of steps, calculated recursively By comparing the minimum value of the three cases plus the current distance To update , ,or Path selection; recursion in the calculation to reach The minimum cumulative distance is terminated, where , and then get the estimated phase , which represents the sequence In the average sequence The position in Represents the index in the cumulative distance calculation process, which is used to traverse and accumulate the distance from the starting point to the current point.

8. The method according to claim 7, characterized in that S500 includes: Assume that at time step t, the action sequence for the next 3 steps output by the Bayesian interaction primitive is , and at this time t>2, and the historical action sequences output at t-1 and t-2 are and , then the current execution action is executed by weighted average of the corresponding actions of the current output action sequence and the historical output action sequence, that is: ； If t<2, then ,After waiting for the robot to complete the action, it continues to execute S400 and S500. In the perception-prediction-execution cycle, the system continues to run until the phase estimation task is completed.

Images