Single-teaching non-rigid object operation skill learning system and method

Abstract

The invention discloses a single-teaching non-rigid object operation skill learning system and method. The system comprises a standard canonical space state conversion module used for converting observation data of a non-rigid target object from any deformation state to a unified standard canonical state; and the standard canonical space coding module is used for constructing a spatial feature field by using the target point cloud data in the standard canonical state. The method comprises the following steps: constructing a non-rigid object operation skill learning system; collecting system training data; pre-training the system; skill learning; and generalizing skills. The method is based on a machine learning algorithm and an advanced sensing technology, so that the robot can quickly and accurately learn the operation skill of the non-rigid object. According to the method, the robot can learn and be applied to different non-rigid object operation tasks by only performing single teaching, so that the teaching time and the labor cost are reduced, and the flexibility and the efficiency of operation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a system and method for learning non-rigid object manipulation skills through single teaching. Background Art

[0002] Traditional methods for learning robot manipulation skills usually require multiple teaching and repeated training, especially when handling non-rigid objects. Non-rigid objects, such as soft objects or deformable objects, have shapes and characteristics that change during manipulation, which places higher demands on learning robot manipulation skills. However, traditional methods require a large amount of teaching and training data and have difficulty adapting to the complexity of non-rigid objects.

[0003] In addition, traditional non-rigid object manipulation methods usually rely on manual modeling and control strategies, which limits the flexibility and adaptability of robots in actual operations. Manual modeling requires expertise and complex processes, and cannot cope with the uncertainty of object deformation. Therefore, a new method is needed to overcome these limitations and enable robots to learn and perform non-rigid object manipulation skills more effectively. In order to solve the above problems, robot operation skill learning methods based on deep learning and reinforcement learning have emerged in recent years. Deep learning technology can be used to model and identify the shape and deformation of objects, thereby providing more accurate operation information. Reinforcement learning technology can enable robots to gradually improve and optimize operation strategies through feedback and optimization. However, these methods still require a large amount of teaching data and training time, and there are still certain challenges in learning the operation of non-rigid objects.

[0004] In the prior art, there is a robot visual motion skill learning method and system (patent number CN117325174A). This method can only learn skills for a specific object. When the target object is replaced with a new object that belongs to the same category but has a different geometric shape, the learned skills become invalid and need to be re-learned for the new target. The existing robot operation skill learning method is targeted at rigid targets and cannot solve the operation skill learning of non-rigid targets. Summary of the invention

[0005] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide a single-teaching non-rigid object manipulation skill learning system and method. The method aims to solve the problem of complex and time-consuming teaching process in traditional robot manipulation skill learning. The proposed method is based on machine learning algorithms and advanced perception technology, enabling robots to quickly and accurately learn non-rigid object manipulation skills.

[0006] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0007] A single-teaching non-rigid object manipulation skill learning system, the system comprising: Standard canonical space state conversion module: used to convert the observation data of non-rigid target objects from arbitrary deformation states to a unified standard canonical space state; The standard canonical space state is a unified state predefined for non-rigid target objects of the same type; for different non-rigid target objects of the same type with different geometric shapes, they are placed on a same support carrier under the influence of gravity and then left to rest naturally. In this case, the deformation state displayed by the non-rigid objects can be defined as a standard canonical space state; Standard canonical space encoding module: uses the target point cloud data in the standard canonical state to construct the spatial feature field; The standard canonical space state conversion module receives the observed point cloud of the non-rigid target object in an arbitrary deformation state as input data, first converts the point cloud data of the non-rigid target object into the standard canonical space state, and then inputs the point cloud data in the standard canonical space state obtained by conversion into the standard canonical space encoding module, thereby constructing the spatial feature field of the non-rigid target object, and using the spatial feature field as the output data of the system.

[0008] Further, the standard canonical space state module includes a point cloud feature extraction unit, a contrast learning unit and a point cloud coordinate prediction unit; The point cloud feature extraction unit takes the observed point cloud of the non-rigid target object as input, and outputs a feature vector with a dimension greater than three for each point encoding in the observed point cloud of the non-rigid target object; The contrastive learning unit takes the point-by-point feature vector output by the point cloud feature extraction unit as input and outputs the corresponding contrastive learning loss; The point cloud coordinate prediction unit divides the standard canonical space into a 64×64×64 spatial grid, and uses its corresponding feature vector to perform sixty-four classifications on the three dimensions of x, y, and z for each point in the input target object observation point cloud, thereby realizing the standard canonical space coordinate prediction of each point in the observation point cloud of the non-rigid target object; The standard canonical space is a cubic space occupied by a non-rigid target object in a standard canonical space state, the center of the cubic space is the center of mass of the non-rigid target object, and the side length of the cubic space is the maximum distance occupied by the non-rigid target object in the three directions of x, y, and z.

[0009] Furthermore, the point cloud feature extraction unit includes: Point cloud sampling and grouping unit: using the farthest point sampling method, a series of key points are selected from the input observation point cloud of the non-rigid target object; then, with each selected key point as the sphere center, all points in the spherical space of the observation point cloud of the non-rigid target object at a specified radius distance are divided into the same group, thereby realizing the sampling and grouping of the observation point cloud of the non-rigid target object; One-dimensional convolution layer: Perform convolution operation on each group divided by the point cloud sampling grouping unit. Each point in the group is convolved with a convolution kernel to obtain a feature vector larger than three dimensions. The convolution kernel size is 1×3. Pooling layer: Using the maximum value selection strategy, the feature vector output by the one-dimensional convolution layer is reduced in dimension to obtain a reduced-dimensional vector; Fully connected layer: fully connects the dimension reduction vector output by the pooling layer to obtain the key feature vector; Upsampling interpolation layer: Upsampling interpolation calculation is performed on the key feature vectors. The number of vectors obtained after sampling is consistent with the number of points input by the point cloud sampling grouping unit.

[0010] Furthermore, the point cloud coordinate prediction unit includes three three-layer fully connected units, which respectively predict coordinates in the three dimensions of x, y, and z; the prediction result is the predicted coordinates of each point in the observed point cloud of the non-rigid target object after being converted to the standard canonical space state, that is, the standard canonical space predicted coordinates, corresponding to the predicted target point; A single three-layer fully connected unit includes three fully connected layers connected in sequence, which performs fully connected processing on the point-by-point feature vector in the observation point cloud of the non-rigid target object, and finally outputs a sixty-four-dimensional vector, and uses the label of the maximum value as the predicted value of one dimension in the three-dimensional coordinate.

[0011] Further, the standard canonical space encoding module includes a point cloud feature propagation unit and a three-dimensional grid reconstruction unit; The standard canonical space encoding module first establishes a three-dimensional space feature vector matrix of a×a×a (for example, a=64), and evenly distributes the feature vectors in the three-dimensional space feature vector matrix in the standard canonical space. The initial value of the feature vector in the three-dimensional space feature vector matrix is ​​zero; the point-by-point feature vectors output by the point cloud feature extraction unit are accumulated to the corresponding positions of the three-dimensional space feature vector matrix according to the predicted coordinates of the standard canonical space to obtain the three-dimensional space feature matrix after mapping; the point cloud feature propagation unit performs convolution operation on the three-dimensional space feature matrix after mapping, so that the features at different spatial positions can interact with each other in context; The point cloud feature propagation unit includes a three-dimensional convolution layer, which performs convolution on the three-dimensional spatial feature matrix after the mapping is completed by using convolution (such as 3×3×3 convolution), and outputs a convolution-processed three-dimensional spatial feature matrix with a dimension greater than three; The three-dimensional grid reconstruction unit uses the three-dimensional space feature matrix after convolution processing to predict the grid enclosing number of each coordinate point in the standard canonical space; the prediction result is used for grid reconstruction; All the intermediate features of the three-dimensional grid reconstruction unit in the process of predicting the grid encirclement number are spliced ​​according to the coordinate position to construct a spatial feature field of the same size as the standard canonical space; The three-dimensional grid reconstruction unit comprises: Feature sampling unit: according to the coordinates of the predicted target point, weighted feature value sampling is performed according to the coordinates in the three-dimensional feature space matrix; Fully connected unit: The sampled feature values ​​are fully connected and finally a floating point number is obtained as the predicted value of the grid enclosing number.

[0012] The non-rigid object manipulation skill learning method of a single teaching includes the following steps: S1. Construct a non-rigid object manipulation skill learning system; S2. System training data collection: In order to obtain data for pre-training the non-rigid object manipulation skill learning system, training data is collected in a virtual simulation environment to obtain a training data set; Select a class of objects as non-rigid target objects, collect mesh data of no less than 20 non-rigid target objects of the same class with different geometric shapes on the Internet, and simplify, reconstruct and complete the mesh data of the collected non-rigid target objects; then load them into Pybullet with set material parameters for simulation data collection; in the simulation environment, initialize each non-rigid target object with a random posture above a desktop and let it fall freely under the action of gravity; in this process, continuously capture its RGBD data when it is in different deformation states; cumulatively capture a set number of frame data for all non-rigid target object models; for each different non-rigid target object, simulate the state it reaches after being placed on the same support carrier under the influence of gravity only as the standard canonical space state; then use the correspondence between the vertices of the mesh of the non-rigid target object in the deformation state and the standard canonical space state to generate the true value of the coordinates of each point on the non-rigid target object in the standard canonical space and the true value of the mesh encirclement number of each point on the non-rigid target object in the standard canonical space state; S3, system pre-training: the observed point cloud data of the non-rigid target object is input into the input end of the non-rigid object manipulation skill learning system. In the standard canonical space state module, the point cloud feature extraction unit extracts a high-dimensional feature for each point in the point cloud; the contrast learning unit calculates all point features to construct the contrast learning loss; the point cloud coordinate prediction unit takes each point feature as input, and predicts the corresponding standard canonical space coordinates point by point, and calculates the prediction loss according to the corresponding true value; in the standard canonical space encoding module, first construct a three-dimensional feature matrix, and input it into the point cloud feature propagation unit for convolution processing; then use the three-dimensional grid reconstruction unit to predict the grid enclosure number of the coordinate point in the canonical space, and construct the reconstruction loss in combination with the true value; then update the system's policy parameters according to the contrast loss, coordinate prediction loss and reconstruction loss to obtain the pre-trained non-rigid object manipulation skill learning system; S4. Skill learning: During the skill demonstration, the observation point cloud in the given key operation steps is , and the corresponding robot end position T, which are input into the pre-trained non-rigid object manipulation skill learning system; observe the point cloud The standard canonical space conversion module and the standard canonical space encoding module are used to construct the standard space feature neural field. ; The end position T of the robot arm is converted to the standard space through the standard canonical space conversion module, which is expressed as ; Transform the end position of the robot arm to the standard space state In the neural feature field, it is encoded into the corresponding feature vector ; S5. Skill generalization: When the skill is applied to a new non-rigid target object, the observed target point cloud is also input into the pre-trained non-rigid object manipulation skill learning system; the point cloud data is finally constructed through the standard canonical space conversion module and the standard canonical space encoding module to construct the standard space feature neural field corresponding to the new non-rigid target object; the feature vector of the end arm posture encoded and recorded in the skill learning is used to perform feature matching in the space feature neural field of the new non-rigid target object, so as to obtain the end arm posture for the new non-rigid target object; according to the end arm posture obtained from the new non-rigid target object in the key operation steps, the execution actions and paths are planned in sequence to complete the skill operation.

[0013] Furthermore, in step S3, the contrastive learning loss It is expressed as: , in, Any observation point cloud in the training data set The The feature vectors obtained by the point cloud feature extraction module are and are any other observation point cloud in the training dataset The jth point in The feature vector obtained by extracting the points through the point cloud feature extraction module; The Points and The jth point in has the same standard canonical space state coordinates, The Points and The kth point in has different standard canonical space state coordinates; express Any point in The set of all point pairs formed by any point in , for Any one of The points do not have the same standard canonical space state coordinate values ​​and The Point pairs consisting of points; is the proportional factor constant; The predicted loss It is expressed as: , in, , , is the true value of the coordinates of any point in the point cloud P in the standard canonical space, , , is the prediction probability of the three-dimensional coordinate value of the point by the coordinate prediction unit; The reconstruction loss It is expressed as: , in, To predict the true value of the grid enclosing number at the target point, The predicted value of the grid encirclement number at the predicted target point; The method of updating the policy parameters of the non-rigid object manipulation skill learning system according to the contrast loss, prediction loss, and reconstruction loss is expressed as: , in, are the parameters of the non-rigid object manipulation skill learning system before updating, The parameters of the learning system for the non-rigid object manipulation skill are updated based on all losses, is the internal learning rate, The gradient is calculated for all losses together, Based on and observation point cloud Contrastive learning loss, Based on and observation point cloud The prediction loss is Based on and observation point cloud reconstruction losses; It is the sum of the three losses for all samples in a batch of training.

[0014] Furthermore, in step S4, the robot end position during the skill learning process is transformed into the standard canonical space by the following method: Given the observation point cloud P and the corresponding robot end position T, the point cloud set of the observed target clamped by the parallel gripper at the end of the robot is recorded as And take their coordinate set in the standard canonical space as , using the least squares method to calculate the displacement rotation matrix between two coordinate sets ; At this time, you can use the formula The end position of the robot arm is converted to the standard canonical space state; where: is the first point in the observation point cloud P that is held by the gripper. Points, Yes The coordinates are transformed into the standard canonical space state; It is the transformation matrix that transforms the part of the observed point cloud P that is gripped by the gripper from the deformed state at the time of observation to the state in the standard canonical space.

[0015] Furthermore, in step S4, using the neural feature field Encode the end-arm pose H converted to the standard space state as a feature vector The method is as follows: Given a 3D mesh model M at the end of the robotic arm, N points are sampled on the 3D mesh model M using the uniform sampling method to form a query point cloud. ; Place the query point cloud G in the standard canonical space according to the robot end posture H converted to the standard space state. The point cloud set at this time is represented as ; For point cloud collection Each coordinate point in is input into the grid number prediction unit of the standard canonical space encoding module, and the intermediate feature vector generated for the point is concatenated to obtain the encoding feature vector of the point; The encoded feature vectors generated by all points constitute a set denoted as , that is, the encoding of the posture using the neural feature field is completed; Represents a point cloud set consisting of N three-dimensional coordinate points.

[0016] Furthermore, in step S5, the method of using the neural feature field to perform pose matching on the new non-rigid target object is as follows: Given a new non-rigid target object’s observation point cloud , and then input them into the standard canonical space conversion module and the standard canonical space encoding module to construct the neural feature field for the new non-rigid target object ; Using query point cloud , randomly initialize a pose , the point cloud By pose Move to this location to get , and use the observation point cloud of the new non-rigid target object to be input at this time The 3D reconstruction unit pair Generate a coded feature vector for each point in to get a set of coded feature vectors ;optimization and Optimize the difference Until convergence, at this time The obtained value is the end operation posture of the new non-rigid target object obtained by matching under the standard state; The process is expressed as: , in, The teaching grasping posture is generalized to generalize the grasping posture in the teaching process to the generalized grasping posture on the new non-rigid target object according to the observed shape and deformation state of the new non-rigid target object.

[0017] Compared with the prior art, the present invention has the following beneficial effects and advantages: 1. Compared with the traditional method, the present invention only needs one teaching, which greatly reduces the number and time of teaching, and greatly improves the efficiency and convenience of operation learning; 2. This method does not rely on manual modeling and control strategies, and can adapt to the complexity and uncertainty of non-rigid objects. The robot can flexibly adjust the operation strategy according to the shape and deformation of the actual object, thereby improving the accuracy and success rate of the operation; 3. The proposed method can be widely used in various industrial and service fields, such as logistics, medical treatment, home assistants, etc., providing an efficient and intelligent method for robots to learn manipulation skills when handling non-rigid objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a structural schematic diagram of a non-rigid object manipulation skill learning system for single teaching in an embodiment of the present invention; Figure 2 A schematic diagram of constructing an observation point cloud of a target object in an embodiment of the present invention; Figure 3 Schematic diagram of the skill learning process in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Note that the following embodiments are merely illustrative in nature, and the present invention is not intended to limit its use or purpose, and the present invention is not limited to the following embodiments.

[0020] The non-rigid object manipulation skill learning system of the single teaching of this embodiment, such as Figure 1 As shown, the system includes: Standard canonical space state conversion module: used to convert the observation data of non-rigid target objects from arbitrary deformation states to a unified standard canonical space state; The standard canonical space state is a unified state predefined for non-rigid target objects of the same type; for different non-rigid target objects of the same type with different geometric shapes, they are placed on a same support carrier under the influence of gravity and then left to rest naturally. In this case, the deformation state displayed by the non-rigid objects can be defined as a standard canonical space state; In one embodiment, a cap is taken as an example: for any cap, the center of mass of its mesh model is moved to the center of mass position of the unit space body, and then the process of wearing it on the head is simulated; finally, the entire cap is scaled so that the length of its largest side is equal to the side length of the unit space body; at this time, the state of the cap's mesh model is the standard canonical space state; Standard canonical space encoding module: uses the target point cloud data in the standard canonical state to construct the spatial feature field; The standard canonical space state conversion module receives the observed point cloud of the non-rigid target object in an arbitrary deformation state as input data, first converts the point cloud data of the non-rigid target object into the standard canonical space state, and then inputs the point cloud data in the standard canonical space state obtained by conversion into the standard canonical space encoding module, thereby constructing the spatial feature field of the non-rigid target object, and using the spatial feature field as the output data of the system.

[0021] The standard canonical space state module described in this embodiment includes a point cloud feature extraction unit, a contrast learning unit, and a point cloud coordinate prediction unit; The point cloud feature extraction unit takes the observed point cloud of the non-rigid target object as input, and outputs a feature vector with a dimension greater than three for each point encoding in the observed point cloud of the non-rigid target object; The contrastive learning unit takes the point-by-point feature vector output by the point cloud feature extraction unit as input and outputs the corresponding contrastive learning loss; The point cloud coordinate prediction unit divides the standard canonical space into a 64×64×64 spatial grid, and uses its corresponding feature vector to perform sixty-four classifications on the three dimensions of x, y, and z for each point in the input target object observation point cloud, thereby realizing the standard canonical space coordinate prediction of each point in the observation point cloud of the non-rigid target object; The standard canonical space described in this embodiment is a cubic space occupied by a non-rigid target object in a standard canonical space state, the center of the cubic space is the center of mass of the non-rigid target object, and the side length of the cubic space is the maximum distance occupied by the non-rigid target object in the three directions of x, y, and z.

[0022] The point cloud feature extraction unit in this embodiment includes: Point cloud sampling and grouping unit: using the farthest point sampling method, a series of key points are selected from the input observation point cloud of the non-rigid target object; then, with each selected key point as the sphere center, all points in the spherical space of the observation point cloud of the non-rigid target object at a specified radius distance are divided into the same group, thereby realizing the sampling and grouping of the observation point cloud of the non-rigid target object; One-dimensional convolution layer: Perform convolution operation on each group divided by the point cloud sampling grouping unit. Each point in the group is convolved with a convolution kernel to obtain a feature vector larger than three dimensions. The convolution kernel size is 1×3. Pooling layer: Using the maximum value selection strategy, the feature vector output by the one-dimensional convolution layer is reduced in dimension to obtain a reduced-dimensional vector; Fully connected layer: fully connects the dimension reduction vector output by the pooling layer to obtain the key feature vector; Upsampling interpolation layer: Upsampling interpolation calculation is performed on the key feature vectors. The number of vectors obtained after sampling is consistent with the number of points input by the point cloud sampling grouping unit.

[0023] Furthermore, the point cloud coordinate prediction unit includes three three-layer fully connected units, which respectively predict coordinates in the three dimensions of x, y, and z; the prediction result is the predicted coordinates of each point in the observed point cloud of the non-rigid target object after being converted to the standard canonical space state, that is, the standard canonical space predicted coordinates, corresponding to the predicted target point; A single three-layer fully connected unit includes three fully connected layers connected in sequence, which performs fully connected processing on the point-by-point feature vector in the observation point cloud of the non-rigid target object, and finally outputs a sixty-four-dimensional vector, and uses the label of the maximum value as the predicted value of one dimension in the three-dimensional coordinate.

[0024] The standard canonical spatial encoding module described in this embodiment includes a point cloud feature propagation unit and a three-dimensional grid reconstruction unit; The standard canonical space encoding module first establishes a 64×64×64 three-dimensional space feature vector matrix, and evenly distributes the feature vectors in the three-dimensional space feature vector matrix in the standard canonical space. The initial value of the feature vector in the three-dimensional space feature vector matrix is ​​zero; the point-by-point feature vector output by the point cloud feature extraction unit is accumulated to the corresponding position of the three-dimensional space feature matrix according to the standard canonical space prediction coordinates to obtain the three-dimensional space feature matrix after mapping; the point cloud feature propagation unit performs convolution operation on the three-dimensional space feature matrix after mapping, so that the features at different spatial positions can interact with each other in context; The point cloud feature propagation unit includes a three-dimensional convolution layer, which uses a 3×3×3 convolution kernel to convolve the mapped three-dimensional spatial feature matrix and outputs a convolution-processed three-dimensional spatial feature matrix with a dimension greater than three; The three-dimensional grid reconstruction unit uses the three-dimensional space feature matrix after convolution processing to predict the grid enclosing number of each coordinate point in the standard canonical space; the prediction result is used for grid reconstruction; All intermediate features of the three-dimensional grid reconstruction unit in the process of predicting the grid enclosing number are spliced ​​according to the coordinate position to construct a spatial feature field of the same size as the standard canonical space.

[0025] The three-dimensional grid reconstruction unit comprises: Feature sampling unit: according to the coordinates of the predicted target point, weighted feature value sampling is performed according to the coordinates in the three-dimensional feature space matrix; Fully connected unit: The sampled feature values ​​are fully connected and finally a floating point number is obtained as the predicted value of the grid enclosing number.

[0026] The single-teaching non-rigid object manipulation skill learning method of this embodiment includes the following steps: S1. Construct a non-rigid object manipulation skill learning system.

[0027] S2. System training data collection: In order to obtain data for pre-training the non-rigid object manipulation skill learning system, training data is collected in a virtual simulation environment to obtain a training data set; A class of objects is selected as non-rigid target objects, and mesh data of no less than twenty non-rigid target objects belonging to the same class with different geometric shapes are collected on the Internet, and the collected mesh data of non-rigid target objects are simplified, reconstructed, and completed; then they are loaded into Pybullet with set material parameters for simulation data collection; in the simulation environment, for each non-rigid target object, it is initialized above a desktop with a random posture and allowed to fall freely under the action of gravity; in this process, its RGBD data in different deformation states is continuously captured; a set number of frame data are cumulatively captured for the models of all non-rigid target objects; for each different non-rigid target object, the state reached after it is placed on the same support vehicle under the influence of gravity only is simulated as the standard canonical space state; then the correspondence between the vertices of the mesh of the non-rigid target object in the deformation state and the standard canonical space state is used to generate the true value of the coordinates of each point on the non-rigid target object in the standard canonical space and the true value of the mesh enclosure number of each point on the non-rigid target object in the standard canonical space state.

[0028] In one embodiment, Figure 2 As shown in the figure, a large amount of mesh data of duckbill caps was first collected from the Internet, and they were simplified, reconstructed, and other processes were performed. Then they were loaded into Pybullet with appropriate material parameters for simulation data collection. In the simulation environment, for each virtual duckbill cap, it was initialized above a desktop with a random posture and allowed to fall freely under the action of gravity. In this process, its RGBD data in different deformation states was continuously captured. A total of 4,000 frames of data were captured for all duckbill cap models. For each different duckbill cap, it was simulated to be worn on the model's head as a standard canonical state. Then the vertex correspondence of the mesh was used to generate the true coordinate value of each point on the duckbill cap in the standard canonical space and the true value of the mesh encirclement number of each point in the standard canonical space.

[0029] S3, system pre-training: the observed point cloud data of the non-rigid target object is input into the input end of the non-rigid object manipulation skill learning system. In the standard canonical space state module, the point cloud feature extraction unit extracts a high-dimensional feature for each point in the point cloud; the contrast learning unit calculates all point features to construct the contrast learning loss; the point cloud coordinate prediction unit takes each point feature as input, and predicts the corresponding standard canonical space coordinates point by point, and calculates the prediction loss according to the corresponding true value; in the standard canonical space encoding module, first construct a three-dimensional feature matrix, and input it into the point cloud feature propagation unit for convolution processing; then use the three-dimensional grid reconstruction unit to predict the grid enclosure number of the coordinate point in the canonical space, and construct the reconstruction loss in combination with the true value; then update the system's policy parameters according to the contrast loss, coordinate prediction loss and reconstruction loss to obtain the pre-trained non-rigid object manipulation skill learning system; The contrastive learning loss It is expressed as: , in, Any observation point cloud in the training data set The The feature vectors obtained by the point cloud feature extraction module are and are any other observation point cloud in the training dataset The jth point in The feature vector obtained by extracting the points through the point cloud feature extraction module; The Points and The jth point in has the same standard canonical space state coordinates, The Points and The kth point in has different standard canonical space state coordinates; express Any point in The set of all point pairs formed by any point in , for Any one of The points do not have the same standard canonical space state coordinate values ​​and The Point pairs consisting of points; is the proportional factor constant; The predicted loss It is expressed as: , in, , , is the true value of the coordinates of any point in the point cloud P in the standard canonical space, , , is the prediction probability of the three-dimensional coordinate value of the point by the coordinate prediction unit; The reconstruction loss It is expressed as: , in, To predict the true value of the grid enclosing number at the target point, The predicted value of the grid encirclement number at the predicted target point; The method of updating the policy parameters of the non-rigid object manipulation skill learning system according to the contrast loss, prediction loss, and reconstruction loss is expressed as: , in, are the parameters of the non-rigid object manipulation skill learning system before updating, The parameters of the learning system for the non-rigid object manipulation skill are updated based on all losses, is the internal learning rate, The gradient is calculated for all losses together, Based on and observation point cloud Contrastive learning loss, Based on and observation point cloud The prediction loss is Based on and observation point cloud reconstruction losses; It is the sum of the three losses for all samples in a batch of training.

[0030] S4. Skill learning: Figure 3 As shown in Figure 2, during the skill demonstration, the observation point cloud of a given key operation step , and the corresponding robot end position T, which are input into the pre-trained non-rigid object manipulation skill learning system; observe the point cloud The standard canonical space conversion module and the standard canonical space encoding module are used to construct the standard space feature neural field. ; The end position T of the robot arm is converted to the standard space through the standard canonical space conversion module, which is expressed as ; Transform the end position of the robot arm to the standard space state In the neural feature field, it is encoded into the corresponding feature vector ; The robot end position during skill learning is transformed into the standard canonical space by the following method: Given the observation point cloud P and the corresponding robot end position T, the point cloud set of the observed target clamped by the parallel gripper at the end of the robot is recorded as And take their coordinate set in the standard canonical space as , using the least squares method to calculate the displacement rotation matrix between two coordinate sets ; At this point, you can use the formula The end position of the robot arm is converted to the standard canonical space state; where: is the first point in the observation point cloud P that is held by the gripper. Points, Yes The coordinates are transformed into the standard canonical space state; It is the transformation matrix that transforms the part of the observed point cloud P that is gripped by the gripper from the deformed state at the time of observation to the state in the standard canonical space.

[0031] Using Neural Feature Fields Encode the end-arm pose H converted to the standard space state as a feature vector The method is as follows: Given a 3D mesh model M at the end of the robotic arm, N points are sampled on the 3D mesh model M using the uniform sampling method to form a query point cloud. ; Place the query point cloud G in the standard canonical space according to the robot end posture H converted to the standard space state. The point cloud set at this time is represented as ; For point cloud collection Each coordinate point in is input into the grid number prediction unit of the standard canonical space encoding module, and the intermediate feature vector generated for the point is concatenated to obtain the encoding feature vector of the point; The encoded feature vectors generated by all points constitute a set denoted as , that is, the encoding of the posture using the neural feature field is completed; Represents a point cloud set consisting of N three-dimensional coordinate points.

[0032] S5. Skill generalization: When the skill is applied to a new non-rigid target object, the observed target point cloud is also input into the pre-trained non-rigid object manipulation skill learning system; the point cloud data is finally constructed through the standard canonical space conversion module and the standard canonical space encoding module to construct the standard space feature neural field corresponding to the new non-rigid target object; the feature vector of the end arm posture encoded and recorded in the skill learning is used to perform feature matching in the space feature neural field of the new non-rigid target object, so as to obtain the end arm posture for the new non-rigid target object; according to the end arm posture obtained from the new non-rigid target object in the key operation steps, the execution actions and paths are planned in sequence to complete the skill operation.

[0033] The method of using neural feature fields to match the pose of new non-rigid target objects is as follows: Given a new non-rigid target object’s observation point cloud , and then input them into the standard canonical space conversion module and the standard canonical space encoding module to construct the neural feature field for the new non-rigid target object ; Using query point cloud , randomly initialize a pose , the point cloud By pose Move to this location to get , and use the observation point cloud of the new non-rigid target object to be input at this time The 3D reconstruction unit pair Generate a coded feature vector for each point in to get a set of coded feature vectors ;optimization and Optimize the difference Until convergence, at this time The obtained value is the end operation posture of the new non-rigid target object obtained by matching under the standard state; The process is expressed in the formula: , in, The teaching grasping posture is generalized to generalize the grasping posture in the teaching process to the generalized grasping posture on the new non-rigid target object according to the observed shape and deformation state of the new non-rigid target object.

[0034] In one embodiment, the application process of the single-teaching non-rigid object manipulation skill learning method is as follows: 1) Build the experimental platform: A corresponding camera (RealSense D435i) is configured at the operating end of the UR5 robot to collect robot visual information, a model head model is placed at a designated position on the workbench as a target for wearing a hat, and a peaked cap is randomly placed on the table as an operating target. At the same time, a set of scenes that are exactly the same as this is built in the simulation environment.

[0035] 2) Collect experimental training data: In order to obtain data for pre-training the non-rigid object manipulation skill learning system, training data was collected in a virtual simulation environment. In order to collect data, a large amount of mesh data of duckbill caps was collected from the Internet, and they were simplified, reconstructed, and other processes were performed. Then they were loaded into Pybullet with the set material parameters for simulation data collection. In the simulation environment, for each virtual duckbill cap, it was initialized above a desktop with a random posture and allowed to fall freely under the action of gravity. In this process, its RGBD data in different deformation states was continuously captured. A total of 4,000 frames of data were captured for all duckbill cap models, and the point cloud of the duckbill cap in the deformation state can be extracted from each frame of data. . Where N is the number of points in the extracted point cloud. Preferably, the number of selected points in this example is 6000. For each different cap, simulate putting it on the model's head as a standard canonical state. Then use the vertex correspondence of the grid to generate the true value of the coordinates of each point on the cap in the standard canonical space. And the true value of the grid enclosing number of each point in the standard canonical space. Preferably, in this example, the standard canonical space unit is divided into a three-dimensional grid space of 128×128×128 dimensions. The grid enclosing number of each small space is calculated based on its centroid coordinates. When any coordinate is given in practical applications, the true value of the grid enclosing number is obtained by interpolation based on the three-dimensional grid. ; When collecting skill teaching data, for the teaching duckbill cap, a demonstration of wearing a duckbill cap is completed by dragging the robotic arm. In each key step, first observe and record the point cloud of the teaching duckbill cap in the state of the step, then drag the end of the robotic arm to the corresponding artificial posture, and record the end posture at this time.

[0036] 3) Build the system: Based on the experimental tasks, construct a non-rigid object manipulation skill learning system.

[0037] 4) If Figure 1 As shown in FIG, the pre-training data collected from the virtual simulation environment, that is, the point cloud data P of the duckbill cap in different deformation states, is input into the input end of the non-rigid object manipulation skill learning system, and firstly input into the standard state conversion module, as shown in FIG. Figure 1As shown, the point cloud feature extraction unit extracts features one by one for each coordinate point in the point cloud P. All point features are then input into the contrastive learning unit to calculate the contrastive learning loss: , in, Any observation point cloud in the training data set The The feature vectors obtained by the point cloud feature extraction module are and are any other observation point cloud in the training dataset The jth point in The feature vector obtained by extracting the points through the point cloud feature extraction module; The Points and The jth point in has the same standard canonical space state coordinates, The Points and The kth point in has different standard canonical space state coordinates; express Any point in The set of all point pairs formed by any point in , for Any one of The points do not have the same standard canonical space state coordinate values ​​and The Point pairs consisting of points; is a constant of the proportional factor; all point features extracted from the point cloud P are input into the coordinate prediction unit, which predicts the coordinate position of each point in a standard canonical space, and calculates the prediction loss in combination with the corresponding true value of the standard canonical space coordinate: , in, , , is the true value of the coordinates of any point in the point cloud P in the standard canonical space, , , is the prediction probability of the three-dimensional coordinate value of the point by the coordinate prediction unit.

[0038] 5) All the point features in the point cloud P obtained in advance by the point cloud feature extraction unit are input into the standard canonical space encoding module according to the standard canonical space coordinate prediction results, and the point cloud feature broadcasting module performs feature convolution processing, and then inputs it into the 3D reconstruction unit for mesh encirclement number prediction and 3D reconstruction. The 3D reconstruction loss is constructed using the mesh encirclement number prediction value and the true value of the mesh encirclement number of each point in the 3D grid space: , in, To predict the true value of the grid enclosing number at the target point, The predicted value of the grid encirclement number at the predicted target point.

[0039] 6) Update the internal parameters of the non-rigid object manipulation skill learning system jointly according to the sum of the three losses.

[0040] 7) Repeat steps 4)-6) until the system is stable, and obtain the pre-trained non-rigid object manipulation skill learning system.

[0041] 8) Given a skill demonstration for wearing a peaked cap, extract three key operation steps, namely: Step 1: Grab the cap; Step 2: Move the cap to the back of the head model; Step 3: Put the cap on your head; In each step, the robot is first used to collect the point cloud data of the cap at that time. , and then manually drag the end of the robot arm to move to the corresponding operation posture and record it as .

[0042] 9) Learn the postures of the three operation steps in sequence: First, the recorded point cloud data Input to the input end of the system, through the standard canonical space conversion module, to obtain the point cloud Point-by-point high-dimensional features and point-by-point standard canonical space coordinates in; Record the gripper in position The point cloud between the two parallel grippers is And their corresponding standard canonical space coordinates are , and use the least squares method to calculate the spatial transformation matrix between them, recorded as ;right Perform space transformation to obtain the operating position of the end of the robot arm in the standard canonical space in this operation step ; Then, the point features are input into the standard canonical space encoding module according to the predicted standard canonical space coordinates, and the feature propagation unit performs point cloud sampling on the end of the robotic arm and records it as point cloud. . Transform G into the pose in the standard canonical space In this state, the terminal point cloud is recorded as , and use the 3D reconstruction unit to predict the 3D grid encirclement number for each point, and splice all the intermediate features generated by the 3D reconstruction unit in this process as the coding feature of the point; The encoding feature set of all points in is recorded as . This completes the learning and coding of the operating postures in the operation steps.

[0043] 10) Given a new target cap n, perform the three steps of the wearing skill accordingly: Collect the point cloud of the target cap , and pass it to the input end of the system, and obtain the standard canonical state point cloud through the standard canonical space conversion module , and the corresponding point-by-point high-dimensional features. The point features are input into the standard canonical space encoding module according to the standard canonical state coordinates. First, a high-dimensional three-dimensional feature grid is constructed through the point cloud feature propagation unit. Then the terminal point cloud G obtained in step 9) is transformed into a random pose , moved to the standard canonical space, recorded as point cloud Similarly, the 3D reconstruction unit is used to predict the grid enclosing number for each point in the point cloud, and the intermediate features are concatenated into coded features. The set of all coded features is recorded as .

[0044] Constructing matching loss Iterative Optimization , so that the matching loss is minimized. Until the loss converges stably, This is the optimal standard canonical state robot end operation posture for the new target cap n in operation step i. Similarly, the point cloud set between the end parallel claws is used to convert the end posture from the standard canonical state to the observation state, that is, the robot end posture in operation step i is obtained. The robot arm plans the operation path and operates the cap according to the operation postures in the three key operation steps, thus completing the generalization application of the wearing skill to the new target cap.

[0045] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the scope of protection of the present invention will not be limited to the embodiments shown herein.

Claims

1. A single-time teaching non-rigid object manipulation skill learning system, characterized in that: The system comprises: Standard canonical space state conversion module: used to convert the observation data of non-rigid target objects from arbitrary deformation states to a unified standard canonical space state; The standard canonical space state is a unified state predefined for non-rigid target objects of the same type; for different non-rigid target objects of the same type with different geometric shapes, they are placed on the same support carrier under the influence of gravity and then left to rest naturally. In this case, the deformation state displayed by the non-rigid objects is defined as the standard canonical space state; Standard canonical space encoding module: uses the target point cloud data in the standard canonical state to construct the spatial feature field; The standard canonical space state conversion module receives the observed point cloud of the non-rigid target object in an arbitrary deformation state as input data, first converts the point cloud data of the non-rigid target object into the standard canonical space state, and then inputs the point cloud data in the standard canonical space state obtained by conversion into the standard canonical space encoding module, thereby constructing the spatial feature field of the non-rigid target object, and using the spatial feature field as the output data of the system.

2. The single-teaching non-rigid object manipulation skill learning system according to claim 1 is characterized in that: The standard canonical space state module includes a point cloud feature extraction unit, a contrast learning unit and a point cloud coordinate prediction unit; The point cloud feature extraction unit takes the observed point cloud of the non-rigid target object as input, and outputs a feature vector with a dimension greater than three for each point encoding in the observed point cloud of the non-rigid target object; The contrastive learning unit takes the point-by-point feature vector output by the point cloud feature extraction unit as input and outputs the corresponding contrastive learning loss; The point cloud coordinate prediction unit divides the standard canonical space into a×a×a spatial grids, and for each point in the input target object observation point cloud, uses its corresponding feature vector to classify the three dimensions of spatial rectangular coordinates x, y, and z, thereby realizing the standard canonical space coordinate prediction of each point in the observation point cloud of the non-rigid target object; The standard canonical space is a cubic space occupied by a non-rigid target object in a standard canonical space state, the center of the cubic space is the center of mass of the non-rigid target object, and the side length of the cubic space is the maximum distance occupied by the non-rigid target object in the three directions of x, y, and z.

3. The single-teaching non-rigid object manipulation skill learning system according to claim 2, characterized in that: The point cloud feature extraction unit includes: Point cloud sampling and grouping unit: using the farthest point sampling method, a series of key points are selected from the input observation point cloud of the non-rigid target object; then, with each selected key point as the sphere center, all points in the spherical space of the observation point cloud of the non-rigid target object at a specified radius distance are divided into the same group, thereby realizing the sampling and grouping of the observation point cloud of the non-rigid target object; One-dimensional convolution layer: Perform convolution operation on each group divided by the point cloud sampling grouping unit, and each point in the group is convolved with the convolution kernel to obtain a feature vector larger than three dimensions; Pooling layer: Using the maximum value selection strategy, the feature vector output by the one-dimensional convolution layer is reduced in dimension to obtain a reduced-dimensional vector; Fully connected layer: fully connects the dimension reduction vector output by the pooling layer to obtain the key feature vector; Upsampling interpolation layer: Upsampling interpolation calculation is performed on the key feature vectors. The number of vectors obtained after sampling is consistent with the number of points input by the point cloud sampling grouping unit.

4. The single-time teaching non-rigid object manipulation skill learning system according to claim 2 is characterized in that: The point cloud coordinate prediction unit includes three three-layer fully connected units, which predict coordinates in the three dimensions of x, y, and z respectively; the prediction result is the predicted coordinates of each point in the observed point cloud of the non-rigid target object after being converted to the standard canonical space state, that is, the standard canonical space prediction coordinates, corresponding to the predicted target point; A single three-layer fully connected unit includes three fully connected layers connected in sequence, performs fully connected processing on the point-by-point feature vector in the observation point cloud of the non-rigid target object, and finally outputs an a-dimensional vector, and uses the label of the maximum value as the predicted value of one dimension in the three-dimensional coordinates.

5. The single-teaching non-rigid object manipulation skill learning system according to claim 4 is characterized in that: The standard canonical space encoding module includes a point cloud feature propagation unit and a three-dimensional grid reconstruction unit; The standard canonical space encoding module first establishes a three-dimensional space feature vector matrix of a×a×a, and evenly distributes the feature vectors in the three-dimensional space feature vector matrix in the standard canonical space. The initial value of the feature vector in the three-dimensional space feature vector matrix is ​​zero; the point-by-point feature vector output by the point cloud feature extraction unit is accumulated to the corresponding position of the three-dimensional space feature vector matrix according to the predicted coordinates of the standard canonical space, and the three-dimensional space feature matrix after mapping is obtained; The point cloud feature propagation unit performs convolution operation on the three-dimensional spatial feature matrix after mapping; The point cloud feature propagation unit includes a three-dimensional convolution layer, which uses a convolution kernel to convolve the mapped three-dimensional spatial feature matrix and outputs a convolution-processed three-dimensional spatial feature matrix with a dimension greater than three; The three-dimensional grid reconstruction unit uses the three-dimensional space feature matrix after convolution processing to predict the grid enclosing number of each coordinate point in the standard canonical space; All the intermediate features of the three-dimensional grid reconstruction unit in the process of predicting the grid encirclement number are spliced ​​according to the coordinate position to construct a spatial feature field of the same size as the standard canonical space; The three-dimensional grid reconstruction unit comprises: Feature sampling unit: according to the coordinates of the predicted target point, weighted feature value sampling is performed according to the coordinates in the three-dimensional feature space matrix; Fully connected unit: The sampled feature values ​​are fully connected and finally a floating point number is obtained as the predicted value of the grid enclosing number.

6. A single-teaching non-rigid object manipulation skill learning method, characterized in that: The following steps are involved: S1. Constructing a non-rigid object manipulation skill learning system for single teaching as claimed in any one of claims 1 to 5; S2. System training data collection: Collect training data in a virtual simulation environment to obtain a training data set; A class of objects is selected as non-rigid target objects, and mesh data of no less than twenty non-rigid target objects of the same class with different geometric shapes are collected from the Internet, and the collected mesh data of non-rigid target objects are simplified, reconstructed, and completed; then loaded into Pybullet with set material parameters for simulation data collection; in the simulation environment, each non-rigid target object is initialized above a desktop with a random posture, and the non-rigid target object is allowed to fall freely under the action of gravity; in this process, its RGBD data in different deformation states is continuously captured; a set number of frame data are cumulatively captured for the models of all non-rigid target objects; for each different non-rigid target object, the state reached after it is placed on the same support carrier under the influence of gravity only is simulated as the standard canonical space state; then the correspondence between the vertices of the mesh of the non-rigid target object in the deformation state and the standard canonical space state is used to generate the true value of the coordinates of each point on the non-rigid target object in the standard canonical space and the true value of the mesh encirclement number of each point on the non-rigid target object in the standard canonical space state; S3, system pre-training: the observed point cloud data of the non-rigid target object is input into the non-rigid object manipulation skill learning system. In the standard canonical space state module, the point cloud feature extraction unit extracts a high-dimensional feature for each point in the point cloud; The contrastive learning unit calculates all point features to construct the contrastive learning loss; the point cloud coordinate prediction unit takes each point feature as input, predicts the corresponding standard canonical space coordinates point by point, and calculates the prediction loss according to the corresponding true value; in the standard canonical space encoding module, first construct a three-dimensional feature matrix and input it to the point cloud feature propagation unit for convolution processing; then use the three-dimensional grid reconstruction unit to predict the grid enclosure number of the coordinate point in the canonical space, and construct the reconstruction loss in combination with the true value; then update the system's strategy parameters according to the contrast loss, coordinate prediction loss and reconstruction loss to obtain a pre-trained non-rigid object manipulation skill learning system; S4. Skill learning: During the skill demonstration, the observation point cloud in the given key operation steps is , and the corresponding robot end position T, and input it into the pre-trained non-rigid object manipulation skill learning system; observe the point cloud The standard canonical space conversion module and the standard canonical space encoding module are used to construct the standard space feature neural field. ; The end position T of the robot arm is converted to the standard space through the standard canonical space conversion module, which is expressed as ; Transform the end position of the robot arm to the standard space state In the neural feature field, it is encoded into the corresponding feature vector ; S5. Skill generalization: When the skill is applied to a new non-rigid target object, the observed target point cloud is also input into the pre-trained non-rigid object manipulation skill learning system; the point cloud data is finally constructed through the standard canonical space conversion module and the standard canonical space encoding module to construct the standard space feature neural field corresponding to the new non-rigid target object; the feature vector of the end arm posture encoded and recorded in the skill learning is used to perform feature matching in the space feature neural field of the new non-rigid target object, so as to obtain the end arm posture for the new non-rigid target object; according to the end arm posture obtained from the new non-rigid target object in the key operation steps, the execution actions and paths are planned in sequence to complete the skill operation.

7. The single-teaching non-rigid object manipulation skill learning method according to claim 6, characterized in that: In step S3, the contrastive learning loss It is expressed as: , in, Any observation point cloud in the training data set The The feature vectors obtained by the point cloud feature extraction module are and are any other observation point cloud in the training dataset The jth point in The feature vector obtained by extracting the points through the point cloud feature extraction module; The Points and The jth point in has the same standard canonical space state coordinates, The Points and The kth point in has different standard canonical space state coordinates; express Any point in The set of all point pairs formed by any point in , for Any one of The points do not have the same standard canonical space state coordinate values ​​and The Point pairs consisting of points; is the proportional factor constant; The predicted loss It is expressed as: , in, , , is the true value of the coordinates of any point in the point cloud P in the standard canonical space, , , is the prediction probability of the three-dimensional coordinate value of the point by the coordinate prediction unit; The reconstruction loss It is expressed as: in, To predict the true value of the grid enclosing number at the target point, The predicted value of the grid encirclement number at the predicted target point; The method of updating the policy parameters of the non-rigid object manipulation skill learning system according to the contrast loss, prediction loss, and reconstruction loss is expressed as: in, are the parameters of the non-rigid object manipulation skill learning system before updating, The parameters of the learning system for the non-rigid object manipulation skill are updated based on all losses, is the internal learning rate, The gradient is calculated for all losses together, Based on and observation point cloud Contrastive learning loss, Based on and observation point cloud The prediction loss is Based on and observation point cloud reconstruction losses; It is the sum of the three losses for all samples in a batch of training.

8. The single-teaching non-rigid object manipulation skill learning method according to claim 6, characterized in that: In step S4, the robot end position during the skill learning process is transformed into the standard canonical space by the following method: Given the observation point cloud P and the corresponding robot end position T, the point cloud set of the observed target clamped by the parallel gripper at the end of the robot is recorded as And take their coordinate set in the standard canonical space as , using the least squares method to calculate the displacement rotation matrix between two coordinate sets ; At this point, we can use the formula The end position of the robot arm is converted to the standard canonical space state; where: is the first point in the observation point cloud P that is held by the gripper. Points, Yes The coordinates are transformed into the standard canonical space state; It is the transformation matrix that transforms the part of the observed point cloud P that is gripped by the gripper from the deformed state at the time of observation to the state in the standard canonical space.

9. The single-teaching non-rigid object manipulation skill learning method according to claim 6, characterized in that: In step S4, the neural feature field is used Encode the end-arm pose H converted to the standard space state as a feature vector The method is as follows: Given a 3D mesh model M at the end of the robotic arm, N points are sampled on the 3D mesh model M using the uniform sampling method to form a query point cloud. ; Place the query point cloud G in the standard canonical space according to the robot end posture H converted to the standard space state. The point cloud set at this time is represented as ; For point cloud collection Each coordinate point in is input into the grid number prediction unit of the standard canonical space encoding module, and the intermediate feature vector generated for the point is concatenated to obtain the encoding feature vector of the point; The encoded feature vectors generated by all points constitute a set denoted as , that is, the encoding of the posture using the neural feature field is completed; Represents a point cloud set consisting of N three-dimensional coordinate points.

10. The single-teaching non-rigid object manipulation skill learning method according to claim 6, characterized in that: In step S5, the method of using the neural feature field to perform pose matching on the new non-rigid target object is as follows: Given a new non-rigid target object’s observation point cloud , and then input them into the standard canonical space conversion module and the standard canonical space encoding module to construct the neural feature field for the new non-rigid target object ; Using query point cloud , randomly initialize a pose , the point cloud By pose Move to this location to get , and use the observation point cloud of the new non-rigid target object to be input at this time The 3D reconstruction unit pair Generate a coded feature vector for each point in to get a set of coded feature vectors ;optimization and Optimize the difference Until convergence, at this time The obtained value is the end operation posture of the new non-rigid target object obtained by matching under the standard state; The process is expressed in the formula: ， in, The teaching grasping posture is generalized to generalize the grasping posture in the teaching process to the generalized grasping posture on the new non-rigid target object according to the observed shape and deformation state of the new non-rigid target object.

Images