Robot cell microoperation skill learning method based on video demonstration

Abstract

The invention discloses a robot cell microoperation skill learning method based on video demonstration. The method comprises the following steps: firstly, identifying two end effectors and a manipulated object in a microoperation video of a robot under a microscope by utilizing a multi-task observation network, and obtaining a teaching space-time trajectory; then, space-time constraints of robot actions are obtained through a task parameterization hidden Markov model; in order to solve the problems of safety and flexibility of robot microoperation at the same time, a soft actor-commentator network based on imitation learning optimization is provided, so that a robot can perform skill learning through demonstration and exploration. According to the method, complex cell manipulation tasks can be executed in a real physical environment, and the development of robot skill learning is expected to be promoted.

Description

Technical Field

[0001] The invention relates to a method for learning cell micro-manipulation skills of a robot in the field of robot technology and biological cell manipulation, and in particular to a method for enabling a robot to learn cell micro-manipulation skills through video demonstration. Background Art

[0002] In the field of micro- and nanoscale automation, robotic micromanipulation technology is one of the key technologies and is widely used in micro-assembly of mechanical parts, biological sample processing, and microsurgery. Especially for fragile and deformable micro-objects such as biological cells, it is particularly challenging to achieve micron-level cell micromanipulation with coordinated operation of multiple end effectors, because cells are easily deformed when subjected to force and excessive deformation may cause irreversible damage. Traditional micromanipulation methods have problems such as large parameter differences, difficulty in flexibly controlling multiple end effectors, and possible damage to cells.

[0003] Therefore, it is necessary to propose a method for learning robot cell micromanipulation skills with a high degree of automation. Summary of the invention

[0004] The present invention aims to provide a robot cell micromanipulation skill learning method based on video demonstration to overcome the shortcomings of the prior art, enable the robot to learn cell micromanipulation skills from videos, improve the safety, flexibility, accuracy, success rate and automation of operations, and reduce the requirements for operators and experimental costs. The present invention enables the robot to learn cell micromanipulation skills from demonstration videos through an imitation learning (IL) framework, which is an imitation learning method inspired by the human learning process.

[0005] The technical solution adopted by the present invention is:

[0006] 1. A video-based method for learning robotic cell micromanipulation skills

[0007] Step 1: Input the robot cell micromanipulation video into the multi-task observation network, and the network outputs the teaching spatiotemporal trajectory;

[0008] Step 2: Use the task parameterized hidden Markov model to obtain the spatiotemporal constraints of the robot action in the teaching spatiotemporal trajectory, and use behavioral cloning to model the strategy reference distribution;

[0009] Step 3: Obtain real-time operation image frames, use the multi-task observation network to obtain the micro-operation state information of the real-time image frames, and then, based on the micro-operation state information of the real-time image frames, the soft actor-critic network optimized by imitation learning explores and learns skills through the spatiotemporal constraints of robot actions and the strategy reference distribution, and outputs the control amount of the robot, thereby controlling the robot;

[0010] Step 4: Repeat step 3 and continue interactive control of the robot until the cell micromanipulation skills are learned.

[0011] In the step one, the multi-task observation network includes a backbone network, a target detection branch, a cell mask segmentation branch and a key point detection branch. The input of the multi-task observation network is used as the input of the backbone network. The backbone network is respectively connected to the target detection branch, the cell mask segmentation branch and the key point detection branch. The target detection branch is used to identify the end effector of the micro-manipulation robot, the cell mask segmentation branch is used to generate a cell mask, and the key point detection branch is used to identify the cell posture.

[0012] The total loss of the multi-task observation network during training is the weighted sum of object detection loss, instance segmentation loss, and keypoint detection loss.

[0013] In the soft actor-critic network based on imitation learning optimization, the actual strategy π is constructed by combining the divergence loss φ The loss function of the learning process is as follows:

[0014]

[0015] in, is the loss function of the policy function, is the expectation, α is the weight estimated by the balanced policy gradient function, π φ (|) is the probability of the policy function, π z is the strategy reference distribution, Q θ () is the action value function, s i and a i are the current state of the agent in the environment and the state s i Select the action you want to take. is the weight of the divergence loss, D KL (||) is the divergence value, is the divergence loss.

[0016] In the soft actor-critic network based on imitation learning optimization, the action a of the micromanipulator i It consists of the moving direction and micro-operation speed. At the initial frame, the spatiotemporal constraints of the robot action are taken as the actual strategy π φ The constrained state space, the exploration ability threshold parameter λ0 and the constraint parameter λ s Satisfying λ0<λ s <1, if an unreasonable state appears in the current state information, a penalty is imposed to avoid potential damage to the cell; in the learning process of subsequent frames, if an unreasonable state appears in the learning process, a penalty is imposed to avoid potential damage to the cell. If the constraint parameter of the round t of exploration ability When it is less than the threshold parameter λ0, the unreasonable state is merged with the current constraint state space to update the constraint state space, so that the constraint state space of the strategy is gradually and dynamically released. At the same time, the divergence loss gradually decreases during the learning process, so that the soft actor-critic network optimized based on imitation learning gradually learns micro-operation skills through exploration.

[0017] The reward function of the soft actor-critic network based on imitation learning optimization satisfies the following formula:

[0018] r i =r base，i +r task,i

[0019]

[0020] r inject，i =-2||e 2，i -e 3，i || 2

[0021]

[0022]

[0023] The subscript i represents the current time, r i is the total reward function, r base，i is the basic reward function, r task，i is the task reward function of different tasks, task = inject, adjust or composite, e 2，i is the position of the injection needle, m c，i and m c，i-1 are the cell areas at the current moment and the previous moment respectively, e 1，i is the position of the suction needle, std() is the standard deviation, N steps is the number of operation steps, r inject，i is the reward function for the needle puncture operation, e 3，i is the position of the cell center, r adjust，i is the reward function for cell posture adjustment, is the current direction of the cell, ω odesire is the target direction of the cell, r composite，i is the reward function of the composite task, c1 is the penalty coefficient for executing tasks in an incorrect order, c2 is the reward coefficient for successful cell posture adjustment, |||| 2 is the square of the vector magnitude.

[0024] 2. A robotic cell micromanipulation skill learning system based on video demonstration

[0025] An image acquisition unit, used for acquiring image frames of robot cell micromanipulation;

[0026] A micro-manipulation state unit, used to obtain state information in the collected robot cell micro-manipulation image frames using a multi-task observation network;

[0027] A teaching space-time trajectory generation unit is used to sequentially input each image frame in the robot cell micro-operation video into the micro-operation state unit to obtain the teaching space-time trajectory;

[0028] A robot action spatiotemporal constraint and strategy reference distribution generation unit, which is used to obtain the robot action spatiotemporal constraint in the teaching spatiotemporal trajectory by using the task parameterized hidden Markov model, and to model the strategy reference distribution by using behavior cloning;

[0029] The skill exploration and learning unit is used to explore and learn skills based on the state information corresponding to the real-time operation image frame obtained by the micro-operation state unit, and output the control amount of the robot and control the robot through the robot's action spatiotemporal constraints and strategy reference distribution based on the soft actor-critic network optimized by imitation learning.

[0030] 3. A computer device

[0031] The device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for learning robot cell micro-manipulation skills based on video demonstration are implemented.

[0032] 4. A computer-readable storage medium

[0033] The medium stores a computer program, which, when executed by a processor, implements the steps of a method for learning robot cell micro-manipulation skills based on video demonstration.

[0034] 5. A computer program product

[0035] The product includes a computer program / instruction, which, when executed by a processor, implements the steps of a method for learning robot cell micro-manipulation skills based on video demonstration.

[0036] The beneficial effects of the present invention are:

[0037] The method proposed in the present invention enables the robot to learn cell micromanipulation skills from videos. By designing a multi-task observation network, the teaching spatiotemporal trajectories of multiple end effectors and operated objects are extracted from expert videos, and the video sequence is converted into an observation sequence containing a variety of information, providing accurate data support for subsequent operations.

[0038] The use of task-parameterized hidden Markov models to extract spatiotemporal constraints in expert demonstrations can effectively identify implicit constraints in micromanipulation tasks, ensure the safety and dexterity of robot operations, and avoid excessive damage to cells.

[0039] The soft actor-critic network based on imitation learning optimization combines demonstration and exploration, enabling the robot to perform complex cell manipulation tasks in actual physical environments, improving the success rate and efficiency of the manipulation.

[0040] Compared with existing methods and manual remote manipulation, the framework of the present invention shortens operation time and reduces cell deformation, which helps to promote the development of robot skill learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0042] Figure 1 It is a flow chart of a method for learning robot cell micro-manipulation skills based on video demonstration according to a preferred embodiment of the present invention;

[0043] Figure 2 It is a schematic diagram of the structure of the multi-task observation network;

[0044] Figure 3 It is a schematic diagram of the convergence process of the loss function of the multi-task observation network;

[0045] Figure 4 It is a schematic diagram of the recognition results of the multi-task observation network;

[0046] Figure 5 It is a schematic diagram of the implicit constraint results identified by compressing, encoding and decoding the spatiotemporal state trajectory through the task parameterized hidden Markov model;

[0047] Figure 6 It is a schematic diagram of the change process of the reward function in the cell manipulation task of the method of the present invention and other methods in the preferred embodiment of the present invention;

[0048] Figure 7 It is a schematic diagram of the process of controlling a robot to complete cell posture adjustment and cell puncture tasks according to the method of the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0050] The multi-DOF micromanipulation robot system used in the experiment includes two robot arms with three degrees of freedom, an inverted microscope system with automatic switching of optical magnification (such as IXplore Standard, OLYMPUS, Japan), a microscope operation platform equipped with an XY motorized stage, and a CMOS camera for image acquisition. The maximum magnification of the microscope eyepiece and objective lens is 400x, and the motion resolution of the robot arm is 0.05μm. The system software and control part are integrated into the Robot Operating System (ROS) Melodic and run on a desktop computer equipped with an Intel Core i9-10980Xe CPU and NVIDIA TITAN Xp, with an operating system of Ubuntu 18.04.

[0051] The experimental subjects were active zebrafish embryonic cells with a diameter of 700-1000μm, the diameter of the holding needle pipette was 800μm, and the diameter of the injection needle was 30μm. Each method was tested 100 times, and the evaluation indicators included whether the holding needle pipette captured the cell, whether the posture adjustment angle was less than 10 degrees, whether the injection needle penetrated the cell, whether the overall operation time was less than 10 seconds, and whether the cell or the end effector was damaged. Success was defined as the cell adjusting to the specified posture and completing the puncture.

[0052] The present invention proposes a method for learning robot cell micromanipulation skills based on video demonstration, such as Figure 1 As shown, the method comprises the following steps:

[0053] Step 1: Several complete robot cell micro-manipulation videos of the same operation type are sequentially input into the multi-task observation network. The network is used to identify the two end effectors and the manipulated object in the robot micro-manipulation video under the microscope, and then output the teaching spatiotemporal trajectory;

[0054] Among them, Figure 2 As shown in FIG. 1 , the multi-task observation network (MTO) includes a backbone network, a target detection branch, a cell mask segmentation branch, and a key point detection branch. The input of the multi-task observation network is used as the input of the backbone network. The backbone network is connected to the target detection branch, the cell mask segmentation branch, and the key point detection branch, respectively. The target detection branch is used to identify the end effectors of the two micro-manipulation robots, the cell mask segmentation branch is used to generate cell masks, and the key point detection branch is used to identify the cell posture. Finally, the position information e1 and e2 of the end effector of each frame, the position e3 of the manipulated cell, the cell posture angle information ω, and the cell mask m are obtained. c Composition teaching space-time trajectory It provides comprehensive state tracking and control basis for multi-task micro-operation, ensures accurate positioning and posture adjustment during micro-operation, and monitors cell deformation to ensure safe operation. The position information e1 and e2 of the end effector are used to track its precise position during the operation; the position e3 of the manipulated cell helps determine the displacement of the cell during the entire operation; the cell posture angle information ω is used to track posture changes to support cell posture adjustment tasks; and the cell mask m c , used for instance segmentation of cells to provide shape and deformation information of cells.

[0055] In this embodiment, the backbone network is ResNeXt. The target detection branch is a region proposal network, which generates candidate target regions and identifies the end effectors of the two micro-manipulators through bounding box regression and category classification. The cell mask segmentation branch is RoIAlign, which generates instance masks on each region of interest to detect cell deformation. The key point detection branch is the Hourglass Network, which generates a key point heat map for identifying cell postures, and finally generates an output heat map through the key point heat map, where each channel represents a key point.

[0056] The total loss of the multi-task observation network during training The loss for object detection Instance segmentation loss and keypoint detection loss The weighted sum of .

[0057] The object detection loss of the multi-task observation network includes the bounding box loss and the category classification loss. The bounding box loss is The category classification loss is in is the bounding box loss, N is the number of bounding boxes, Smooth L1 () is a smooth L1 loss function, which is used to calculate the difference between the predicted bounding box and the true bounding box. is the i-th true bounding box, is the bounding box of the i-th prediction; the instance segmentation loss is Where H and W are the height and width of the segmentation mask, respectively. is the jth pixel value of the i-th true segmentation mask, is the jth pixel value of the i-th predicted segmentation mask; the key point detection loss is in, is the Gaussian distribution value of the kth key point at the jth pixel position of the i-th true heat map, is the Gaussian distribution value of the kth key point at the jth pixel position of the ith predicted heat map, and K is the number of key points.

[0058] When training the MTO network, a multi-task observation network is used to process a video dataset containing robot cell micro-operations. The dataset is divided into an original dataset and a dataset with added noise. During the training process, the convergence of the loss function is recorded, and the trend of object detection loss, instance segmentation loss, and key point detection loss with the training rounds is observed, such as Figure 3 As shown, Figure 3 (a) Figure 3 (b) Figure 3 (c) and Figure 3 (d) are the processes in which the bounding box loss, target classification loss, instance segmentation loss, and key point detection loss decrease and converge as the number of training rounds increases. Among them, the three weights λ obj , mask and λ key Set to 0.3, 0.4 and 0.3 respectively.

[0059] To verify the accuracy and robustness of MTO for multi-task observation, tests were conducted on the original dataset and the dataset with added noise. For the target detection task, the mean average accuracy (mAP) was calculated. When processing the original dataset and the noisy dataset, the target detection mAP (0.5:0.95) was 0.928 and 0.911, respectively, with a change rate of 1.83%. For the instance segmentation task, the mAP was also calculated. The instance segmentation mAP (0.5:0.95) of the original dataset and the noisy dataset were 0.863 and 0.852, respectively, with a change rate of 1.27%. In the posture recognition task, the angle recognition error was measured. The angle recognition errors of the original dataset and the noisy dataset were 1.47° and 1.49°, respectively, with a change rate of 1.36%. The experimental results show that MTO has high accuracy and robustness in multi-task observation. The recognition results of the video frames are shown in the figure. Figure 4 As shown in Figure 2, after the multi-task network recognition, we can obtain the position information e1 and e2 of the end effector, the position e3 of the manipulated cell, the cell posture angle information ω, and the cell mask m. c .

[0060] Several continuous control tasks were selected for testing, including two end effectors approaching the operated object (basic task), cell puncture, cell posture adjustment, and composite tasks of puncture after posture adjustment. Ten demonstration videos were prepared for each task.

[0061] Step 2: Use the task parameterized hidden Markov model (THMM) to obtain the spatiotemporal constraints of the robot action in the teaching spatiotemporal trajectory, and use the behavior cloning modeling strategy reference distribution. The constraint distributions of the cell puncture trajectory and the cell posture adjustment trajectory of the end effector operation trajectory e1 and e2 are as follows: Figure 5 (a) and Figure 5 as shown in (b).

[0062] Specifically:

[0063] The observation sequence is associated with an unknown implicit state sequence, which consists of K discrete hidden states. Different hidden states correspond to different constraints in the robot micro-manipulation task, and the hidden states follow Distribution, a i，j represents the transition probability from state i to state j, and the probability of staying in the state for s consecutive time steps is given by the Gaussian distribution Estimate, observe feature ξ t The multivariate Gaussian distribution from state j, with the population parameter set Πi is the probability of initial state i, a i，m is the probability of transitioning from state i to state m, K is the total number of hidden states, are the mean, covariance, and mean and covariance of the time spent in state i, respectively. The spatiotemporal state trajectories are compressed, encoded, and decoded by THMM to identify implicit constraints in micro-manipulation tasks.

[0064] The compression and encoding process of the spatiotemporal state trajectory is as follows: the expected complete log-likelihood of the THMM is maximized by the expectation maximization (EM) method, thereby estimating the overall parameter set of the THMM.

[0065] The decoding process of the spatiotemporal state trajectory is as follows: use the Viterbi method to decode the most likely hidden state sequence, calculate the probability of the observed sequence in the hidden state, predict the state sequence in the future time range through deterministic sampling, and obtain the spatiotemporal constraint S THMM ={s1, s2, ..., s K}.

[0066] Use the behavior cloning algorithm to obtain the strategy reference distribution π z The process is as follows: the position information of the end effector at the current moment, the position of the manipulated cell, the cell's posture angle information, and the cell mask are used as the state, and the position information of the end effector at the next moment is used as the action. After using the multi-layer perceptron (MLP) for behavior cloning modeling, the strategy reference distribution π corresponding to the action output for any state is obtained. z .

[0067] Step 3: Obtain real-time operation image frames and use the multi-task observation network to obtain the micro-operation state information s of the real-time image frames i ,Then, according to the micro-operation state information of the real-time image frame, the soft actor-critic network based on imitation learning optimization (ILOSAC) explores and learns skills through the spatiotemporal constraints of robot actions and the strategy reference distribution, and outputs the control amount of the robot, thereby controlling the robot;

[0068] Step 4: Repeat step 3 and continue interactive control of the robot until the cell micromanipulation skills are learned.

[0069] The present invention expands the ordinary Markov decision process (MDP) into a spatiotemporal constrained Markov decision process (SCMPD), including a state space S, an action space A, a transition probability P, a reward function R and a cost function C. The action a of the micromanipulator i By moving direction d i and micro-operation speed v i composition, is the moving direction of the left holding needle, is the movement speed of the left holding needle, is the moving direction of the right holding needle, is the movement speed of the right holding needle, |||| is the modulus of the vector, through the strategy π φ (a|s) (parameterized by a Gaussian function with mean μ φ (s) and covariance σ φ (s) is given by a deep neural network (DNN) and the critic function Q θ (s, a) (defined by the DNN to estimate the expected reward of taking an action at state) to make a decision. The critic parameter θ is calculated by minimizing the loss Conduct training, including is the expected value of the state-action pair, y i is the target value used to train the critic network, consisting of the immediate reward and the expected return of the next state, γ is the reward discount factor, is the expected return under strategy π, exploration rate λ and temperature coefficient α, s0 is the initial state, a0 is the initial action, y i = r(s i , a i )+γ(Q θ (s i+1 , a i+1 )-αlog(π φ (a i+1 |s t+1 ))) 2 , α is the temperature coefficient, which is used to control the balance between exploration and utilization is the loss function of the temperature coefficient, From strategy π i The expected value of the sampled action, π i (|) is the strategy π i The action probability distribution under i is the strategy distribution of the strategy in state i, is the minimum entropy objective. Strategy π φ Reference strategy π z Constraint, by minimizing the Kullback-Leibler divergence D KL (π z ||π φ ) optimizes the strategy parameters. Among them, the actual strategy π is constructed by combining the divergence loss φ The loss function of the learning process is as follows:

[0070]

[0071] in, is the loss function of the policy function, which is used to measure the quality of the current policy parameter φ. is the expectation, which represents the distribution sampling of the state, α is the weight of the balanced policy gradient function estimation, π φ (|) is the probability of the policy function, π z is the strategy reference distribution, is the action value function, s i and a i are the current state of the agent in the environment and the state s i Select the action you want to take. is the weight of the divergence loss, λ KL <1, as the interaction time t increases, Gradually decrease, D KL (||) is the divergence value, which is used to calculate the difference between two probability distributions. is the divergence loss.

[0072] Actions of a micromanipulator in a soft actor-critic network optimized by imitation learning i It consists of the moving direction and micro-operation speed. In the initial frame, the robot motion spatiotemporal constraint S THMM As a practical strategy π φ The constrained state space, the exploration ability threshold parameter λ0 and the constraint parameter λ s Satisfying λ0<λ s <1, if an unreasonable state appears in the current state information, that is, the state information s obtained by the multi-task observation network i If the current constraint state space is exceeded, a penalty is imposed to avoid potential damage to the cell; in the learning process of subsequent frames, if an unreasonable state occurs during the learning process, a penalty is imposed to avoid potential damage to the cell. If the constraint parameter of the round t of exploration ability When it is less than the threshold parameter λ0, the unreasonable state is merged with the current constraint state space to update the constraint state space, so that the constraint state space of the strategy is gradually released dynamically. Gradually decreases, so the divergence loss gradually decreases, and the constraints include constrained state space and divergence loss, that is, the constraints are gradually weakened, so that the soft actor-critic network based on imitation learning optimization gradually learns new micro-operation skills through exploration. During the exploration process, the actions of the agent are mainly supervised and constrained by the reward function.

[0073] The reward function of the soft actor-critic network based on imitation learning optimization satisfies the following formula:

[0074] r i =r base,i +r task,i

[0075]

[0076] r inject，i =-2||e 2，i -e 3，i || 2

[0077]

[0078]

[0079] The subscript i represents the current time, r i is the total reward function, r base，i is the basic reward function, r task，i is the task reward function of different tasks, task = inject, adjust or composite, inject, adjust or composite represent the composite tasks of cell puncture, cell posture adjustment and cell posture adjustment followed by cell puncture, respectively, e 2，i is the position of the injection needle, m c，i and m c，i-1 are the cell areas at the current moment and the previous moment respectively, e 1，i is the position of the suction needle, std() is the standard deviation, N steps is the number of operation steps, r inject，i is the reward function for the needle puncture operation, e 3，i is the position of the cell center, r adjust，i is the reward function for cell posture adjustment, is the current direction of the cell, ω desire is the target direction of the cell, r composite，i is the reward function of the composite task, c1 is the penalty coefficient for executing tasks in an incorrect order, which is 9 in this embodiment, c2 is the reward coefficient for successful cell posture adjustment, which is 3 in this embodiment, |||| 2is the square of the vector magnitude.

[0080] -||e 2，i -m c，i || 2 and ||e 1,i -m c，i || 2 They are used to restrain the injection needle and hold the needle pipette close to the cell. Used to constrain cell deformation, -||m c，i -m c,i-1| | 2 Used to prevent cells from deforming greatly in a short period of time, -0.1N steps Used to prevent the operation time from being too long, -||e 2，i -e 3，i || 2 Used to enable the injection needle to reach the center of the cell for puncture operation. Used to make the injection needle rub against the cells to complete posture adjustment. composite，i The injection needle completes the cell posture adjustment and puncture operations in sequence. When performing the posture adjustment task, When performing cell puncture tasks, r composite，i =-3||e 2，i -e 3，i || 2 If the system attempts to pierce before adjusting the posture, r composite，i = -c1 (c1 = 9), when the cell posture matches the target r composite，i =-c2(c2=3).

[0081] The performance of the method of the present invention is compared with other benchmark methods (such as DDPG, SAC, SOIL, DDPGfD, DAPG, etc.). Figure 6 As shown. Among them, Figure 6 (a) is the basic end effector close to the cell learning process, Figure 6 (b) is the learning process of the puncture task. Figure 6 (c) is the learning process of the cell posture adjustment task. Figure 6(d) is the learning process of the compound task of first adjusting the posture and then puncturing. In the basic task, the learning rate and stability of each method are slightly different. The method of the present invention can achieve higher returns, such as -0.32 for the ILOSAC method. In the injection task, the ILOSAC method has obvious advantages, fast learning speed and the highest average return (-0.81), while the SAC and DDPG methods have large learning curve oscillations due to the unconstrained policy gradient. In the adjustment task, guided by imitation learning, the four imitation learning frameworks can obtain higher returns within a short period of interaction, but SAC and DDPG learn from scratch and it is difficult to master effective operation skills in a short time. The average return of the ILOSAC method is significantly better than other methods. For compound tasks, due to the high complexity of the task, the final returns of the DDPG and SAC methods are low and the variance is large. The ILOSAC method quickly acquires skills by applying spatiotemporal constraints, and can learn dexterous skills by weakening constraints and increasing the exploration space, thereby obtaining a higher return than the existing imitation learning methods.

[0082] To prove the superiority and task versatility of the proposed method, verification experiments were conducted on each task in a real environment and the success rate was recorded. The experimental results show that the proposed framework has the highest success rate in multiple tasks, such as 97%, 86%, 77% and 68% in basic tasks, injection tasks, adjustment tasks and composite tasks, respectively, while other methods such as DDPG have success rates of 77%, 56%, 38% and 19% in these tasks, SAC has success rates of 74%, 61%, 45%, SOIL has success rates of 83%, 69%, 58% and 42%, DAPG has success rates of 89%, 81%, 63% and 54%, DDPGfD has success rates of 85%, 75%, 66% and 47% and lower levels. Figure 7 Schematic diagram of the cell posture adjustment and cell puncture process completed by ILOSAC-controlled injection needle and holding needle respectively.

[0083] The present invention also proposes a robot cell micro-manipulation skill learning system based on video demonstration, the system comprising:

[0084] An image acquisition unit, used for acquiring image frames of robot cell micromanipulation;

[0085] A micro-manipulation state unit, used to obtain state information in the collected robot cell micro-manipulation image frames using a multi-task observation network;

[0086] A teaching space-time trajectory generation unit is used to sequentially input each image frame in the robot cell micro-operation video into the micro-operation state unit to obtain the teaching space-time trajectory;

[0087] The robot action spatiotemporal constraint and strategy reference distribution generation unit is used to obtain the robot action spatiotemporal constraint and strategy reference distribution π in the teaching spatiotemporal trajectory using the task parameterized hidden Markov model z ;

[0088] The skill exploration and learning unit is used to obtain the state information corresponding to the real-time operation image frame obtained by the micro-operation state unit, and the soft actor-critic network based on imitation learning optimization is used to obtain the state information corresponding to the real-time operation image frame according to the robot action spatiotemporal constraints and strategy reference distribution π z Explore and learn skills and output the robot's control volume and control the robot.

[0089] The present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a robot cell micro-manipulation skill learning method based on video demonstration are implemented.

[0090] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of a robot cell micro-manipulation skill learning method based on video demonstration are implemented.

[0091] The present invention also proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a method for learning robot cell micro-manipulation skills based on video demonstration.

[0092] Finally, it should be noted that the above embodiments and explanations are only used to illustrate the technical solution of the present invention rather than to limit it. Those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope disclosed in the technical solution of the present invention, which should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for learning robot cell micromanipulation skills based on video demonstration, characterized in that: The following steps are involved: Step 1: Input the robot cell micromanipulation video into the multi-task observation network, and the network outputs the teaching spatiotemporal trajectory; Step 2: Use the task parameterized hidden Markov model to obtain the spatiotemporal constraints of the robot action in the teaching spatiotemporal trajectory, and use behavioral cloning to model the strategy reference distribution; Step 3: Obtain real-time operation image frames, use the multi-task observation network to obtain the micro-operation state information of the real-time image frames, and then, based on the micro-operation state information of the real-time image frames, the soft actor-critic network optimized by imitation learning explores and learns skills through the spatiotemporal constraints of robot actions and the strategy reference distribution, and outputs the control amount of the robot, thereby controlling the robot; Step 4: Repeat step 3 and continue to interactively control the robot until the cell micromanipulation skills are learned.

2. A method for learning robot cell micro-manipulation skills based on video demonstration according to claim 1, characterized in that: In the step one, the multi-task observation network includes a backbone network, a target detection branch, a cell mask segmentation branch and a key point detection branch. The input of the multi-task observation network is used as the input of the backbone network. The backbone network is respectively connected to the target detection branch, the cell mask segmentation branch and the key point detection branch. The target detection branch is used to identify the end effector of the micro-manipulation robot, the cell mask segmentation branch is used to generate a cell mask, and the key point detection branch is used to identify the cell posture.

3. A method for learning robot cell micro-manipulation skills based on video demonstration according to claim 2, characterized in that: The total loss of the multi-task observation network during training is the weighted sum of object detection loss, instance segmentation loss, and keypoint detection loss.

4. The method for learning robot cell micro-manipulation skills based on video demonstration according to claim 1, characterized in that: In the soft actor-critic network based on imitation learning optimization, the actual strategy π is constructed by combining the divergence loss φ The loss function of the learning process is as follows: in, is the loss function of the policy function, is the expectation, α is the weight estimated by the balanced policy gradient function, π φ (|) is the probability of the policy function, π z is the strategy reference distribution, is the action value function, s i and a i are the current state of the agent in the environment and the state s i Select the action you want to take. is the weight of the divergence loss, D KL (||) is the divergence value, is the divergence loss.

5. The method for learning robot cell micro-manipulation skills based on video demonstration according to claim 1, characterized in that: In the soft actor-critic network based on imitation learning optimization, the action a of the micromanipulator i It consists of the moving direction and micro-operation speed. At the initial frame, the spatiotemporal constraints of the robot action are taken as the actual strategy π φ The constrained state space, the exploration ability threshold parameter λ0 and the constraint parameter λ s Satisfies λ0<λ s <1, if an unreasonable state appears in the current state information, a penalty is imposed to avoid potential damage to the cell; in the learning process of subsequent frames, if an unreasonable state appears in the learning process, a penalty is imposed to avoid potential damage to the cell. If the constraint parameter of the round t of exploration ability When it is less than the threshold parameter λ0, the unreasonable state is merged with the current constraint state space to update the constraint state space, so that the constraint state space of the strategy is gradually and dynamically released. At the same time, the divergence loss gradually decreases during the learning process, so that the soft actor-critic network optimized based on imitation learning gradually learns micro-operation skills through exploration.

6. The method for learning robot cell micro-manipulation skills based on video demonstration according to claim 1, characterized in that: The reward function of the soft actor-critic network based on imitation learning optimization satisfies the following formula: r i ＝r base,i +r task,i r inject,i =-2‖e 2,i -And 3,i ‖ 2 The subscript i represents the current time, r i is the total reward function, r base,i is the basic reward function, r task,i is the task reward function of different tasks, task = inject, adjust or composite, e 2,i is the position of the injection needle, m c,i and m c,i-1 are the cell areas at the current moment and the previous moment respectively, e 1,i is the position of the suction needle, std() is the standard deviation, N steps is the number of operation steps, r inject,i is the reward function for the needle puncture operation, e 3,i is the position of the cell center, r adjust,i is the reward function for cell posture adjustment, is the current direction of the cell, ω desire is the target direction of the cell, r composite,i is the reward function of the composite task, c1 is the penalty coefficient for executing tasks in an incorrect order, c2 is the reward coefficient for successful cell posture adjustment, ‖‖ 2 is the square of the vector magnitude.

7. A robot cell micro-manipulation skill learning system based on video demonstration, characterized in that: include: An image acquisition unit, used for acquiring image frames of robot cell micromanipulation; A micro-manipulation state unit, used to obtain state information in the collected robot cell micro-manipulation image frames using a multi-task observation network; A teaching space-time trajectory generation unit is used to sequentially input each image frame in the robot cell micro-operation video into the micro-operation state unit to obtain the teaching space-time trajectory; A robot action spatiotemporal constraint and strategy reference distribution generation unit, which is used to obtain the robot action spatiotemporal constraint in the teaching spatiotemporal trajectory by using the task parameterized hidden Markov model, and obtain the strategy reference distribution by using the behavior cloning algorithm; The skill exploration and learning unit is used to explore and learn skills based on the state information corresponding to the real-time operation image frame obtained by the micro-operation state unit, and output the control amount of the robot and control the robot through the robot's action spatiotemporal constraints and strategy reference distribution based on the soft actor-critic network optimized by imitation learning.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of a method for learning robot cell micro-manipulation skills based on video demonstration as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for learning robot cell micro-manipulation skills based on video demonstration as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of a method for learning robot cell micro-manipulation skills based on video demonstration as described in any one of claims 1 to 6 are implemented.

Images