A human-robot collaborative trajectory imitation learning method based on kernelized motion primitives

Abstract

The present application relates to human-computer collaborative motion trajectory imitation learning field, disclose a kind of human-computer collaborative covering trajectory imitation learning method based on kernelization motion base element, comprising the following steps:1) the human-computer joint demonstration trajectory in human-computer collaborative covering process is collected, and pretreatment is carried out, to build training sample.2) using kernelization motion base element model, learn the nonlinear relationship between hand position and mechanical arm end position in training sample.3) using the fabric grabbing point detection method based on semantic segmentation identifies and grabs cloth corner;4) in prediction stage, according to the hand position detected by RGB-D camera, the response position of mechanical arm end is predicted in real time, generates the end motion trajectory coordinated with the current position of hand and its motion trend for robot, to complete human-computer collaborative covering action.

Description

Technical Field

[0001] This invention relates to the field of human-machine collaborative motion trajectory imitation learning, and specifically to a human-machine collaborative tiled trajectory imitation learning method based on kernelized motion primitives. Background Technology

[0002] Human-robot collaboration, as an important auxiliary service function of robots, has broad development prospects. For example, human-robot collaborative bedding can solve situations where the object to be laid is too large for one person to complete. As a common and typical human-robot collaborative task, in the execution of human-robot collaborative bedding, the robot needs to predict its own response trajectory based on the observed human motion trajectory and interaction strategy to cooperate with the human hand movements to complete the bedding task. However, because the human hand motion trajectory has a certain degree of spontaneity, the target position of the human hand movement often changes, making it difficult to learn the nonlinear mapping relationship between the human hand trajectory and the robot's end effector trajectory through manual programming.

[0003] According to literature reviews, machine learning methods such as supervised learning, unsupervised learning, and reinforcement learning can be used to learn and reproduce robot operation skills. However, the learning process of these methods is based on a large amount of training data, while in actual human-robot collaborative scenarios, the amount of teaching data is usually small. In addition, there are also methods that use imitation learning, which obtain teaching samples by having a teacher demonstrate the human hand trajectory and the robot trajectory. The robot learns this non-linear relationship from the teaching samples and generalizes this relationship to new environments and tasks, effectively improving the robot's ability to learn and generalize human-robot collaborative tasks.

[0004] Existing imitation learning algorithms for human-robot cooperative motion trajectories mainly fall into two categories: time-driven imitation learning algorithms and hand position-driven imitation learning algorithms. Hand position-driven imitation learning algorithms directly predict the corresponding end-effector position, avoiding time alignment issues and overcoming the drawbacks of time-driven imitation learning algorithms' stringent environmental assumptions, thus exhibiting greater applicability. While classic Gaussian Processes (GP) and Gaussian Mixture Models (GMMs) can directly encode the nonlinear relationship between hand and end-effector positions, their lack of extrapolation capabilities makes it difficult to directly adjust existing models to meet new task requirements. Task-Parameterized Gaussian Mixture Models (TP-GMMs) improve the extrapolation properties of Gaussian Mixture Models by modeling local trajectory features, but still struggle to adjust arbitrary trajectory points. Therefore, there is an urgent need to provide a skill learning and reproduction method suitable for human-robot cooperative tasks such as cooperative cabling, to improve the robot's human-robot cooperative capabilities. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to provide a human-machine collaborative laying trajectory imitation learning method based on nuclear motion primitives, which enables robots to learn and reproduce human-machine collaborative laying skills.

[0006] The specific technical solution for achieving the objective of this invention is as follows:

[0007] A human-machine collaborative tile-laying trajectory imitation learning method based on nucleated motion primitives includes the following steps:

[0008] A human-machine collaborative tile-laying trajectory imitation learning method based on nucleated motion primitives is characterized by the following steps:

[0009] Step 1: For the human-machine collaborative paving task, collect multiple human-machine joint teaching trajectory samples in the workspace, and preprocess the collected teaching samples, that is, use Gaussian mixture model and Gaussian mixture regression algorithm to fit an optimal probability reference trajectory described by conditional probability distribution to represent the motion attributes, thereby constructing a training sample set.

[0010] Step 2: Train the nucleated motion primitive model in the training sample set, that is, use the probabilistic reference trajectory to learn the nonlinear relationship between the human hand position and the end position of the robotic arm in the human-machine collaborative laying task.

[0011] Step 3, Fabric grabbing, which involves using a semantic segmentation-based fabric grabbing point detection method to identify and grab a corner of the fabric;

[0012] Step 4: In the prediction phase, the reasoning process of the nucleated motion primitive model predicts the response position of the robotic arm end based on the position of the human hand detected by the RGB-D camera in real time. This generates an end-effector trajectory for the robot that is coordinated with the current position and movement trend of the human hand's covering action, thereby completing the human-machine collaborative covering action.

[0013] Furthermore, the specific method for step 1 is as follows:

[0014] The direct teaching method is used to simultaneously collect the motion trajectories of the human hand and the robot, and then combine the two to form a human-robot joint teaching trajectory. All human-robot joint teaching trajectories are generated by... It means that among them Indicates the input point for the hand position. Indicates the position output point of the robot's end effector. L and H represent the dimension of the input and output spaces, the length of the teaching trajectory, and the number of teaching trajectories, respectively. l and h represent the sequence number of the human-machine joint teaching trajectory to which the human hand position input point and the robot end effector position output point belong, and the sequence number of that point in the trajectory, respectively.

[0015] After collecting multiple human-machine joint teaching trajectory samples, a probabilistic reference trajectory containing the probability characteristics of the samples was fitted from them using Gaussian mixture model and Gaussian mixture regression algorithm. To characterize the motion attributes of an action;

[0016] During trajectory preprocessing, a Gaussian Mixture Model (GMM) is first used to obtain the probability distribution of the taught trajectory, i.e., the joint probability distribution.

[0017]

[0018] Where s represents the hand position input, ξ represents the robot end effector position output, and π c μ c and Σ c Let represent the prior probability, mean, and covariance of the c-th Gaussian component, respectively, where C represents the number of Gaussian components, and π is the inequality in the Gaussian mixture model. c μ c and Σ c The parameters were solved iteratively using the EM algorithm, the initialization of the model parameters was completed using the K-means clustering algorithm, and the optimal value of the number of Gaussian components C was estimated using the BIC criterion.

[0019] Then, a probabilistic reference trajectory is obtained using the Gaussian mixture regression (GMR) algorithm. Where N represents the number of input-output point pairs contained in the probability reference trajectory, let s n This represents the input point representing the position of the i-th person's hand in the probability reference trajectory. s n The estimated value of the position output corresponding to the robot's end effector, for each input point s. n corresponding All are determined by conditional probability distributions Description, in which and Take s respectively from the input n hour The mean and covariance estimates, and the probabilistic reference trajectory, summarize the variability in the human-machine joint teaching trajectory and the correlation between input and output.

[0020] Furthermore, the specific method for step 2 is as follows:

[0021] Using probabilistic reference trajectories M hand position sampling points {s} are extracted from the workspace using random sampling. m , m = 1, 2, ..., M} and calculate the mean value of the corresponding end effector position of the robotic arm. and covariance As training samples for the nucleated motion primitive model, the nonlinear relationship between the human hand position and the end effector position of the robotic arm, as shown in formula (3), is then learned to achieve the new human hand position input s * The corresponding expected position of the robotic arm end effector E(ξ(s)) * The reasoning of )); define the kernel function k(·,·) as the Gaussian kernel Where h and σ f For hyperparameters; define the kernel matrix k(s) i s j )=k(s i s j )I o , where I o Let K be an O-order identity matrix. Further define matrix K as...

[0022]

[0023] Then we can obtain

[0024] E(ξ(s * ))=k * (K+λ∑) -1 μ (11)

[0025] Where k * =[k(s * ,s1)k(s * ,s2)…k(s * s M )], λ is the coefficient of the penalty term introduced to avoid overfitting.

[0026] Furthermore, the specific method for step 3 is as follows:

[0027] First, a semantic segmentation network based on UNet is trained. The edges and corners of a real fabric are sprayed with different colors of paint. Pixel labels are then generated using color recognition. The "outer edge" of the fabric is defined as the area within 1.5cm of the fabric edge, the "inner edge" as the area within 1.5cm of the "outer edge," and the "corner" as the 3*3cm area at the fabric corner. A loss function is defined. For each class of pixels s∈S, the binary cross-entropy loss is... The mean is shown in formula (4):

[0028]

[0029] Where i is a pixel in the input depth image I, w s For the weight of each type of label, y i It is the label of the i-th pixel. The label of the i-th pixel as predicted by the network;

[0030] Secondly, the grasping direction of the grasping points is analyzed. After thresholding the output of the neural network, the values ​​belonging to the "outer edge" E are obtained. O The point set p = [p x p y ]∈E O And the "inner edge" E I The point set q = [q x q y ]∈E I , where p x p y q x q y Let p be the coordinates of the midpoints of the "inner" and "outer" edges on the x and y axes, respectively. For each point p in the "outer" edge, a point q in the "inner" edge can be found. * =[q x q y ]make

[0031]

[0032] Where ||·||2 is the 2-norm, given by p and q ＊ The correspondence between them defines the grabbing direction at point p as from p to q. ＊ ;

[0033] Then, the uncertainty of the grasping direction at the grasping point is analyzed, and N is defined. t (p) represents the distance E from the "outer edge" o Let α be the set of the t nearest pixels to point p in the vector vector α. The angle between the point p and the positive x-axis can be used to obtain the sine and cosine values ​​of the grabbing direction, as shown in formula (6):

[0034]

[0035] Define sine and cosine value vectors x0(p) and x1(p).

[0036]

[0037] The uncertainty U(p) of the grasping direction prediction is defined as the sum of the variances of the sine and cosine vectors, i.e., U(p) = Var(x0(p)) + Var(x1(p)), where Var(x0(p)) and Var(x1(p)) are the variances of x0(p) and x1(p), respectively.

[0038] Choose circle E with a radius of 4cm centered at the center of the fabric "corner" area. circle The point with the minimum uncertainty is the grab point p. * ,Right now

[0039]

[0040] Finally, a sliding gripping strategy is used, in which the gripper moves at a 45° angle to the horizontal x-axis along the gripping direction to the target position and then picks it up.

[0041] Furthermore, the specific method for step 4 is as follows:

[0042] First, an RGB-D camera is used to detect and acquire the real-time position coordinates of the human hand during the human-robot collaborative covering process. Then, these hand coordinates are used as input to a kernelized motion primitive model to infer the corresponding end-effector response position of the robotic arm. Finally, an end-effector trajectory is generated for the robot that coordinates with the current position and movement trend of the human hand during the covering action, thus completing the human-robot collaborative covering action.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] (1) The technical solution of the present invention uses the imitation learning method to learn human-machine collaborative laying skills, avoiding the collection of a large amount of training data, and can obtain motion attributes from a small number of teaching samples and build a stable system model.

[0045] (2) The technical solution of the present invention uses a nucleated motion primitive model to learn the nonlinear relationship between the position of the human hand and the position of the end of the robotic arm. This model can adjust the teaching trajectory to any intermediate path or target point (position and velocity), and has extrapolation capability, thus having better trajectory generalization capability.

[0046] (3) The technical solution of the present invention utilizes a fabric gripping point detection method based on semantic segmentation to identify and grip fabric corners. Using depth images as neural network input, it can achieve gripping of fabrics of different colors and patterns. In addition, compared with traditional fabric gripping point detection methods, this method has a higher gripping success rate. Attached Figure Description

[0047] Figure 1 This is a flowchart of the overall method of the technical solution of the present invention.

[0048] Figure 2This is a human-machine collaborative tiling trajectory imitation learning scenario in an embodiment of the present invention.

[0049] Figure 3 This is a human-machine collaborative teaching trajectory sample in an embodiment of the present invention.

[0050] Figure 4 These are random sampling points of the probability reference trajectory in the embodiments of the present invention.

[0051] Figure 5 This is a schematic diagram of the fabric grabbing process in an embodiment of the present invention.

[0052] Figure 6 The above is a simulation experiment diagram of an embodiment of the present invention. Detailed Implementation

[0053] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0054] This invention provides a human-machine collaborative tile-laying trajectory imitation learning method based on nucleated motion primitives. Figure 1 This is a flowchart illustrating the overall process of the human-machine collaborative tile trajectory imitation learning method based on nucleated motion primitives provided by the present invention. The working scenario of the embodiment is as follows: Figure 2 As shown, it includes the following steps:

[0055] Step 1: For the human-machine collaborative tiling task, multiple human-machine joint teaching trajectory samples are collected in the workspace. The collected teaching samples are preprocessed, that is, a Gaussian mixture model and Gaussian mixture regression algorithm are used to fit an optimal probability reference trajectory described by a conditional probability distribution to represent the motion attributes, thereby constructing a training sample set. Specifically:

[0056] The technical solution of this invention adopts a direct teaching method to simultaneously collect the motion trajectories of the human hand and the robot. The human hand trajectory is obtained by detecting the human hand movement trajectory of an operator through a human hand tracking algorithm. The corresponding collaborative motion trajectory of the robot end effector is obtained by another operator dragging the robot in "gravity compensation" mode. After the human hand and robot end effector trajectories are collected simultaneously, the two are combined to form a human-machine joint teaching trajectory.

[0057] To avoid the randomness and noise in demonstrations caused by single-person and joint teaching trajectories, it is necessary to collect multiple teaching trajectories and preprocess them, that is, to use Gaussian mixture models and Gaussian mixture regression algorithms to fit a probability reference trajectory containing the probability characteristics of the samples. To characterize the motion attributes of the action. In this embodiment, 10 human-machine collaborative teaching trajectories were sampled, and the collected human-machine collaborative teaching trajectory samples are as follows: Figure 3 As shown. All human-machine collaborative teaching trajectories are... It means that, among them Indicates the input point for the hand position. Indicates the position output point of the robot's end effector. L and H represent the dimension of the input and output spaces, the length of the teaching trajectory, and the number of teaching trajectories, respectively; l and h represent the sequence number of the human-machine joint teaching trajectory to which the human hand position input point and the robot end effector position output point belong, and the sequence number of that point in the trajectory, respectively.

[0058] During trajectory preprocessing, a Gaussian Mixture Model (GMM) is first used to obtain the probability distribution of the taught trajectory, i.e., the joint probability distribution.

[0059]

[0060] Where s represents the hand position input, ξ represents the robot end effector position output, and π c μ c and ∑ c Let represent the prior probability, mean, and covariance of the c-th Gaussian component, respectively, where C represents the number of Gaussian components, and π is the inequality in the Gaussian mixture model. c μ c and ∑ c The parameters were solved iteratively using the EM algorithm, the initialization of the model parameters was completed using the K-means clustering algorithm, and the optimal value of the number of Gaussian components C was estimated using the BIC criterion.

[0061] Then, a probabilistic reference trajectory is obtained using the Gaussian mixture regression (GMR) algorithm. Where N represents the number of input-output point pairs contained in the probability reference trajectory. Let s n This represents the input point representing the position of the i-th person's hand in the probability reference trajectory. s n The estimated value of the position output corresponding to the robot's end effector, for each input point s. n corresponding All are determined by conditional probability distributions Description, in which and Take s respectively from the input n hour The mean and covariance estimates are given. The probabilistic reference trajectory summarizes the variability in the human-machine joint teaching trajectory and the correlation between input and output.

[0062] Step 2: Train the kernelized motion primitive model on the training sample set. Specifically, using probabilistic reference trajectories, learn the nonlinear relationship between the human hand position and the robotic arm end effector position in the human-robot collaborative tiling task's joint trajectory samples.

[0063] Using probabilistic reference trajectories M hand position sampling points {s} are extracted from the workspace using random sampling. m , m = 1, 2, ..., M} and calculate the mean value of the corresponding end effector position of the robotic arm. and covariance As training samples for the nucleated motion primitive model, the nonlinear relationship between the human hand position and the end effector position of the robotic arm, as shown in formula (11), is then learned to achieve the new human hand position input s * The corresponding desired position of the robotic arm's end effector. The reasoning is as follows. In this embodiment, the number of sampling points is M = 500.

[0064] Define the kernel function k(·,·) as the Gaussian kernel. Where h and σ f For hyperparameters. Define the kernel matrix k(s) i s j )=k(s i s j )I O , where I O Let K be an O-order identity matrix. Further define matrix K as...

[0065]

[0066] Then we can obtain

[0067] E(ξ(s * ))=k * (K+λ∑) -1 μ (19)

[0068] Where k * =[k(s * ,s1)k(s * ,s2)…k(s * s M )], λ is the coefficient of the penalty term introduced to avoid overfitting. In this embodiment, λ = 1, h = 0.5, and σ f =1.

[0069] Step 3, Fabric Grabbing, involves identifying and grabbing a corner of the fabric using a semantic segmentation-based fabric grabbing point detection method. Specifically:

[0070] First, a semantic segmentation network based on UNet is trained. The edges and corners of a real fabric are sprayed with different colors of paint. Pixel labels are then generated using color recognition. The "outer edge" of the fabric is defined as the area within 1.5cm of the fabric edge, the "inner edge" as the area within 1.5cm of the "outer edge," and the "corner" as the 3*3cm area at the fabric corner. A loss function is defined. For each class of pixels s∈S, the binary cross-entropy loss is... The mean is shown in formula (12):

[0071]

[0072] Where i is a pixel in the input depth image I, w s For the weight of each type of label, y i It is the true label of the i-th pixel. The label of the i-th pixel as predicted by the network;

[0073] Secondly, the grasping direction of the grasping points is analyzed. After thresholding the output of the neural network, the values ​​belonging to the "outer edge" E are obtained. O The point set p = [p x p y ]∈E O And the "inner edge" E I The point set q = [q x q y ]∈E I , where p x p y q x q y These are the x and y coordinates of the midpoints of the "inner" and "outer" edges, respectively. For each point p in the "outer" edge, a point q in the "inner" edge can be found. * =[q x q y ]make

[0074]

[0075] Where ||·||2 is the 2-norm. From p and q ＊ The correspondence between them defines the grabbing direction at point p as from p to q. ＊ ;

[0076] Then, the uncertainty of the grasping direction at the grasping point is analyzed, and N is defined. t (p) represents the distance E from the "outer edge" O Let α be the set of the t nearest pixels to point p in the vector vector α. The angle between the angle and the positive x-axis can be used to obtain the sine and cosine values ​​of the grabbing direction at point p, as shown in formula (14):

[0077]

[0078] Define sine and cosine value vectors x0(p) and x1(p).

[0079]

[0080] The uncertainty U(p) of the grasping direction prediction is defined as the sum of the variances of the sine and cosine vectors, i.e., U(p) = Var(x0(p)) + Var(x1(p)), where Var(x0(p)) and Var(x1(p)) are the variances of x0(p) and x1(p), respectively.

[0081] Choose circle E with a radius of 4cm centered at the center of the fabric "corner" area. circle The point with the minimum uncertainty is the grab point p. * ,Right now

[0082]

[0083] Finally, a sliding gripping strategy is used, in which the gripper is tilted at 45° to the horizontal x-axis and moves along the gripping direction to the target position before gripping.

[0084] Step 4: In the prediction phase, the inference process of the nucleated motion primitive model predicts the response position of the robotic arm's end effector in real time based on the hand position detected by the RGB-D camera. This generates an end effector trajectory for the robot that coordinates with the current position and movement trend of the human hand's covering action, thereby completing the human-robot collaborative covering action. Specifically:

[0085] After the robot completes the task of grasping the fabric corner, it can collaborate with the human to complete the laying task. In the skill reproduction phase, the Kinect V2 RGB-D camera is first used to acquire the position coordinates of the human hand in the camera coordinate system in real time using a KCF-based hand tracking algorithm. Then, the coordinates are transformed to the robot's base coordinate system and used as input to the kernelized motion primitive model to predict the response position of the robotic arm's end effector in real time. Finally, a trajectory planning algorithm is used to generate an end effector trajectory for the robot that is coordinated with the current position and movement trend of the human hand's laying action, thereby completing the human-robot collaborative laying action. Figure 6 The diagram shows the prediction of the robotic arm's end-effector trajectory generated by the human hand trajectory using leave-one-out cross-validation. It can be seen that the technical solution of the present invention can effectively learn and reproduce human-machine collaborative laying skills.

Claims

1. A human-machine collaborative tile-laying trajectory imitation learning method based on nucleated motion primitives, characterized in that, Includes the following steps: Step 1: For the human-machine collaborative paving task, collect multiple human-machine joint teaching trajectory samples in the workspace, and preprocess the collected teaching samples, that is, use Gaussian mixture model and Gaussian mixture regression algorithm to fit an optimal probability reference trajectory described by conditional probability distribution to represent the motion attributes, thereby constructing a training sample set. Step 2: Train the nucleated motion primitive model in the training sample set, that is, use the probabilistic reference trajectory to learn the nonlinear relationship between the human hand position and the end position of the robotic arm in the human-machine collaborative laying task. Step 3, Fabric grabbing, which involves using a semantic segmentation-based fabric grabbing point detection method to identify and grab a corner of the fabric; Step 4: In the prediction phase, the reasoning process of the nucleated motion primitive model predicts the response position of the robotic arm end based on the position of the human hand detected by the RGB-D camera in real time. This generates an end-effector trajectory for the robot that is coordinated with the current position and movement trend of the human hand's covering action, thereby completing the human-machine collaborative covering action.

2. The human-machine collaborative laying trajectory imitation learning method based on nucleated motion primitives according to claim 1, characterized in that, The specific method for step 1 is as follows: The direct teaching method is used to simultaneously collect the motion trajectories of the human hand and the robot, and then combine the two to form a human-robot joint teaching trajectory. All human-robot joint teaching trajectories are generated by... It means that, among them Indicates the input point for the hand position. Indicates the position output point of the robot's end effector. , , , These represent the dimension of the input and output spaces, the length of the teaching trajectory, and the number of teaching trajectories, respectively. and These represent the sequence number of the human-machine joint teaching trajectory to which the human hand position input point and the robot end effector position output point belong, and the sequence number of that point in the trajectory, respectively; After collecting multiple human-machine joint teaching trajectory samples, a probabilistic reference trajectory containing the probability characteristics of the samples was fitted from them using a Gaussian mixture model and a Gaussian mixture regression algorithm. To characterize the motion attributes of an action; During trajectory preprocessing, a Gaussian Mixture Model (GMM) is first used to obtain the probability distribution of the taught trajectory, i.e., the joint probability distribution. ; in, This indicates the input of the hand position. This indicates the position output of the robot's end effector. , and They represent the first Prior probabilities, mean, and covariance of each Gaussian component This represents the number of Gaussian components in a Gaussian mixture model. , and The parameters were solved iteratively using the EM algorithm, and the model parameters were initialized using the K-means clustering algorithm. The number of Gaussian components was determined. Use the BIC criterion to estimate the optimal value; Then, a probabilistic reference trajectory is obtained using the Gaussian mixture regression (GMR) algorithm. ,in This represents the number of input-output point pairs contained in the probability reference trajectory. In the probability reference trajectory, the first Personal hand position input point, express The estimated value of the position output corresponding to the robot's end effector, for each input point. corresponding All are determined by conditional probability distributions Description, in which and Take the input respectively hour The mean and covariance estimates, and the probabilistic reference trajectory, summarize the variability in the human-machine joint teaching trajectory and the correlation between input and output.

3. The human-machine collaborative laying trajectory imitation learning method based on nucleated motion primitives according to claim 2, characterized in that, The specific method for step 2 is as follows: Using probabilistic reference trajectories Extracting from the workspace using random sampling Personal hand position sampling point And calculate the average position of the corresponding robotic arm end effector. and covariance As training samples for the nucleated motion primitive model, the nonlinear relationship between the human hand position and the end effector position of the robotic arm, as shown in formula (3), is then learned to enable the processing of new human hand position inputs. The corresponding desired position of the robotic arm's end effector. The reasoning; Define kernel function Gaussian kernel ,in and Define hyperparameters; define the kernel matrix. ,in for An identity matrix of order 1, further defining the matrix for Then we can obtain in , , , The coefficient of the penalty term introduced to avoid overfitting.

4. The human-machine collaborative laying trajectory imitation learning method based on nucleated motion primitives according to claim 1, characterized in that, The specific method for step 3 is as follows: First, a semantic segmentation network based on UNet is trained. The edges and corners of a real fabric are sprayed with different colors of paint. Pixel labels are then generated using color recognition. The "outer edge" of the fabric is defined as the area within 1.5cm of the fabric edge, the "inner edge" as the area within 1.5cm of the "outer edge," and the "corner" as the 3*3cm area at the fabric corner. A loss function is defined. For each category Binary cross-entropy loss of mid-pixel The mean is shown in formula (4): in Input depth image One pixel in , For the weight of each type of label, It is the first A label of 1 pixel, The first one predicted by the network A label of one pixel; Secondly, the grasping direction of the grasping points is analyzed. After thresholding the output of the neural network, the points belonging to the "outer edge" are identified. point set and "inner edge" point set ,in , , , The midpoints of the "inner and outer edges" are respectively , The coordinates on the axis for each point in the "outer edge" Points in the "inner edge" can be found in all of them. Make in It is a 2-norm, by and The correspondence between them will The grab direction at the point is defined as arrive ; Then, the uncertainty of the grasping direction of the grasping point is analyzed, and a definition is made. Distance from the "outer edge" Points in Recent A set of pixels, let For vectors and The angle between the positive directions of the axis can be obtained. The sine and cosine values ​​of the point grab direction are shown in formula (6): Define sine and cosine value vectors and Define the magnitude of uncertainty in the grasping direction prediction. It is the sum of the variances of the sine and cosine value vectors, i.e. ,in and They are respectively and The variance; Choose a circle with a radius of 4cm centered at the center of the fabric's "corner" area. The point with the least uncertainty is the grab point. ,Right now Finally, a sliding gripping strategy is used, that is, the gripper and the horizontal... The axis is tilted at 45° and moved to the target position along the gripping direction before being gripped.

5. The human-machine collaborative laying trajectory imitation learning method based on nucleated motion primitives according to claim 1, characterized in that, The specific method for step 4 is as follows: First, an RGB-D camera is used to detect and obtain the real-time position coordinates of the human hand during the human-machine collaborative laying process. Then, the position coordinates of the human hand are used as input to the kernelized motion primitive model to infer the corresponding end-effector response position of the robotic arm. The robot generates an end-effector trajectory that is coordinated with the current position and motion trend of the human hand laying action, thereby completing the human-machine collaborative laying action.

Images