Non-rigid 3D object modeling using scene flow estimation

Abstract

An apparatus for reconstructing a non-rigid object comprises a depth sensor (204); a processor configured to repeatedly perform the following steps: capturing a point cloud using the depth sensor, wherein a first point cloud is set as a canonical model; estimating a scene flow (403) using the captured point cloud and a previous point cloud; pre-warping the captured point cloud using the estimated scene flow; estimating a deformation parameter of the captured point cloud; warping the pre-warping point cloud using the deformation parameter; merging the warped point cloud into the canonical model. The apparatus enables accurate and stable modeling of non-rigid objects.

Description

TECHNICAL FIELD

[0001] The present invention relates to a method and apparatus for reconstructing a non-rigid 3D object. BACKGROUND

[0002] 3D object reconstruction using sequential depth images (point clouds) input from camera sensors has been extensively studied in the field of computer vision and robotics. To reconstruct a 3D object from sequential depth images, input depth images are aligned and fused into a single coordinate system. One challenge in this field is non-rigid object reconstruction, in which a reconstruction target moves and deforms throughout a scanning process. In this case, an algorithm needs to handle object surface deformation. To achieve non-rigid 3D object reconstruction, people have proposed methods based on non-rigid iterative closest point (ICP). Non-rigid ICP is a technique that tracks input point clouds, which can handle object surface deformation using as-rigid-as-possible (ARAP) constraints. These methods can be used to track surface deformation of an object throughout a scanning process, and then use tracked surface deformation information to fuse input point clouds into a single coordinate system. However, it is difficult to handle large motion of a target object. This is because a correspondence search process in a deformation tracking process is difficult. Existing methods use a previous state of a target object to find a correspondence between a reconstructed model and input point cloud data. If there is large motion or a large range of occlusion, the correspondence search process fails. Therefore, improving a surface deformation tracking process is an important issue in non-rigid 3D object reconstruction.

[0003] One class of existing methods is non-rigid ICP based on a prior model. A specific template 3D model, such as a face and a human body, is used before tracking a target object in input data. In these methods, first, a template 3D model is fitted to input data by estimating model parameters, such as joint angles, shape parameters, and landmark positions. Then, the estimated model parameters are used to track a target object. In one method, a 3D face model and a body model are used as a prior model to track a target object. In another method, a 3D human body model is generated from input color and depth (RGB-D) images by using a machine learning based human body model, and then the model is used to track a reconstruction target. These methods can be used to stabilize tracking and reconstruct a target object.

[0004] The most important aspect of these methods is that they are specific to concrete objects, such as the human body and face. These methods use 3D prior models to track the target object. Therefore, they cannot be used for arbitrary non-rigid 3D modeling tasks. Although these methods do not explicitly use template models during reconstruction, they require a human model dataset to train the neural network. Therefore, the trained network cannot handle non-human objects. These are the limitations of the methods described above.

[0005] Another existing approach is scene flow estimation, a technique for tracking input 3D point clouds. In these methods, machine learning techniques are used to estimate the 3D flow of the point cloud. In one approach, the 3D scene flow is trained on a 2D image domain. This method outputs a rigidity mask, which is then decomposed into multiple rigid scene flows from the entire rigid scene flow. Therefore, this method can distinguish between camera motion and object motion. In another approach, the scene flow is trained directly in 3D space. The network learns flow embeddings representing point motion. These methods can be applied to general 3D point cloud tracking tasks.

[0006] One problem with existing 3D scene flow estimation methods is their inapplicability to modeling non-rigid objects. These methods do not regularize for surface deformation, thus failing to address flow ambiguity. Furthermore, while the latter approach allows for direct training of networks in 3D space, it cannot handle large point cloud inputs. Additionally, this approach only considers point differences at a single scale. Therefore, it is difficult to estimate fine-grained, dense scene flows using the latter method. These are the limitations of the aforementioned methods.

[0007] In the reconstruction of non-rigid 3D objects caused by large-scale motion of the target object, it is necessary to solve the problem of point cloud tracking. Summary of the Invention

[0008] The present invention provides a device for reconstructing non-rigid objects, so as to accurately and stably reconstruct non-rigid objects.

[0009] According to a first aspect, an apparatus for reconstructing non-rigid objects is provided. The apparatus includes: a depth sensor; and a processor for repeatedly performing the following steps: capturing a point cloud using the depth sensor, wherein a first point cloud is set as a canonical model; estimating a scene flow using the captured point cloud and a previous point cloud; pre-distorting the captured point cloud using the estimated scene flow; estimating deformation parameters of the captured point cloud; distorting the pre-distorted point cloud using the deformation parameters; and merging the distorted point cloud into the canonical model.

[0010] In one possible implementation of the first aspect, the device further includes a display, and the estimated scene flow includes repeatedly performing the following steps before receiving user input to end the following steps: estimating the scene flow using the captured point cloud and the previous point cloud; projecting the estimated scene flow onto a plane and displaying it on the display; and receiving user input to adjust the scene flow estimation parameters.

[0011] In one possible implementation of the first aspect, estimating the scene flow includes: generating a first point feature from the previous point cloud using a first point cloud convolutional kernel, and inputting the first point feature into the next layer of the first point cloud convolutional kernel; generating a second point feature from the captured point cloud using a second point cloud convolutional kernel, and inputting the second point feature into the next layer of the second point cloud convolutional kernel; and estimating the scene flow using the first point feature, the second point feature, and the scene flow estimated in the next layer.

[0012] In one possible implementation of the first aspect, estimating the scene flow using the first point feature and the second point feature, as well as the scene flow estimated in the next layer, comprises: distorting the point feature using an initial scene flow calculated based on the scene flow estimated in the next layer; calculating a 3D cost amount based on a cost that is as rigid as possible; and refining the initial scene flow based on the 3D cost amount to obtain the estimated scene flow.

[0013] According to a second aspect, a method for reconstructing a non-rigid object is provided. The method includes repeatedly performing the following steps: capturing a point cloud using a depth sensor, wherein a first point cloud is set as a canonical model; estimating a scene flow using the captured point cloud and a previous point cloud; pre-distorting the captured point cloud using the estimated scene flow; estimating deformation parameters of the captured point cloud; distorting the pre-distorted point cloud using the deformation parameters; and merging the distorted point cloud into the canonical model.

[0014] According to a third aspect, a storage medium is provided. The storage medium stores a program that, when executed by a processor, causes the processor to perform the method provided in the second aspect. Attached Figure Description

[0015] To more clearly describe the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art are briefly introduced below. The accompanying drawings in the following description only show some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0016] Figure 1A An example of a use case for the reconstruction system 101 provided in the first embodiment is shown.

[0017] Figure 1B An example of a use case for the reconstruction system 102 provided in the first embodiment is shown.

[0018] Figure 2 A block diagram of the hardware configuration provided in the first embodiment is shown.

[0019] Figure 3 A block diagram of the functional configuration provided in the first embodiment is shown.

[0020] Figure 4 A flowchart of non-rigid object reconstruction provided in the first embodiment is shown.

[0021] Figure 5 The overall architecture of the 3D scene flow estimation network is shown.

[0022] Figure 6 The details of the scene flow estimator for estimator-L1 505 are shown.

[0023] Figure 7 This is a flowchart of the scene flow estimation parameter adjustment process.

[0024] Figure 8 An example of a user interface for adjusting 3D scene flow estimation parameters is shown. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should fall within the scope of protection of the present invention.

[0026] Figure 1A and Figure 1B An example of a use case for the reconstruction system 101 / 102 provided in the first embodiment of the present invention is shown. Reconstruction system 101 includes a fixed depth sensor connected to a computer. Reconstruction system 102 includes a movable depth sensor embedded in a mobile device such as a smartphone. A target 100, such as a human, is reconstructed by scanning with the depth sensor. During the scanning process, such as... Figure 1A As shown, the reconstructed target 100 moves in front of a fixed depth sensor to display its overall shape in various directions, or as... Figure 1B As shown, the user holds the depth sensor and moves it around the reconstructed target 100 to scan the reconstructed target 100.

[0027] Figure 2 This is a block diagram of the hardware configuration provided in the first embodiment. The reconstruction system includes a Central Processing Unit (CPU) 200, a Graphics Processing Unit (GPU) 201, Random Access Memory (RAM) 202, Read Only Memory (ROM) 203, a bus 209, an Input / Output Interface (I / F) 205, a display 207, and a user interface 208. The reconstruction system 101 / 102 also includes a depth sensor 204 and a storage device 206, both connected to the bus 209 via the I / F 205. The CPU 200 controls each component connected via the bus 209 by executing programs. The RAM 202 serves as the main memory for the CPU 200, etc. The ROM 203 stores the operating system (OS), programs, device drivers, etc. The depth sensor 204, connected via the I / F 205, captures depth images. In the depth image, depth values ​​are visualized, where the depth value is defined as the distance to the target. The storage device 206 connected via input / output I / F 205 is a mass storage device, such as a hard disk or flash memory. Input / output I / F 205 converts data between the reconstruction system 101 / 102 and the storage device 206. GPU 201 processes and renders the data by executing programs. Display 207 shows the captured depth image, the results of intermediate processes, and the user interface. The user interface 208 embedded in the reconstruction system 101 / 102 receives user input and transmits it to CPU 200.

[0028] Figure 3 This is a block diagram illustrating the functional configuration provided in the first embodiment. When the program is used by CPU 200 and GPU 201... Figure 2 This functional configuration is achieved when the hardware in the system is executed. The reconstruction system 101 / 102 includes a user interface control unit 300, a point cloud capture unit 301, a scene flow estimation unit 302, a scene flow visualization unit 306, a non-rigid ICP unit 303, a point cloud fusion unit 304, and a storage unit 305.

[0029] The user interface control unit 300 processes user input. Based on the user input, the user interface is displayed on the monitor 207.

[0030] The point cloud acquisition unit 301 receives a series of depth images from the depth sensor 204. The received depth images are converted into point clouds.

[0031] The scene flow estimation unit 302 uses the depth sensor 204 to estimate the 3D scene flow of the reconstructed target 100 from the acquired point cloud. The 3D scene flow represents the motion of an object in a 3D space with 3D vectors.

[0032] The scene flow visualization unit 306 makes the estimated scene flow visible on the display 207.

[0033] The non-rigid ICP element 303 estimates the surface deformation and uses the estimated surface deformation parameters to warp the acquired point cloud. The non-rigid ICP element 303 also estimates 3D rigid transformation parameters to align the warped point cloud. The non-rigid ICP unit 303 can be implemented using any depth-based non-rigid ICP algorithm, such as DynamicFusion (e.g., see “DynamicFusion: Reconstruction and Tracking of Non-rigid Scenes in Real-Time” by R. Newcombe, D. Fox, and S. Seitz at the IEEE Conference on Computer Vision and Pattern Recognition in Boston, Massachusetts in 2015) and SurfelWarp (e.g., see “SurfelWarp: Efficient Non-Volumetric Dynamic Reconstruction” by W. Gao and R. Tedrake at Robotic: Science and System in 2018).

[0034] Point cloud fusion unit 304 merges distorted point clouds to reconstruct the 3D model of reconstruction target 101. Point cloud fusion unit 304 can be implemented using any model fusion algorithm, such as model fusion based on truncated signed distance function (TSDF) using voxel representation and surfel-based model fusion using surfel representation.

[0035] Storage unit 305 stores the reconstructed 3D model in storage device 206 for future use.

[0036] Figure 4This is a flowchart of the non-rigid object reconstruction provided in the first embodiment. As described above in conjunction with Figure 1, the user operates the reconstruction system. The user fixes or holds the depth sensor 204 in the reconstruction system 101 / 102 to dynamically scan the reconstruction target 100. The reconstruction target 100 can be any type of non-rigid object, such as a human, a face, and a soft toy. The user should move around the reconstruction target 100, or the reconstruction target 100 should be moved to display its overall shape in various directions in front of the depth sensor 204.

[0037] From a hardware configuration perspective Figure 4 Each step in the process is executed on CPU 200 and GPU 201, and data is stored in RAM 202 or storage device 206 and loaded from it as needed. CPU 200 receives a series of point clouds from depth sensor 204 and stores them in RAM 202. The point clouds captured by depth sensor 204 can be stored in storage device 206. The following describes the functional configuration. Figure 4 Each step in the process.

[0038] In step 400, the point cloud acquisition unit 301 acquires a point cloud from the depth sensor 204 or the storage device 206 storing previously acquired point clouds. Specifically, the point cloud acquisition unit 301 uses the depth sensor 204 to capture a series of depth images of the reconstructed target 100. The acquired depth images are converted into point clouds.

[0039] In step 401, it is checked whether the input point cloud is the first point cloud input. If it is the first point cloud input, then in step 402, the point cloud is set as a canonical model. The canonical model is the basic model, and subsequent point clouds will be merged into this canonical model.

[0040] For frames other than the first frame (the first point cloud), the scene flow estimation unit 302 uses point clouds t and t–1 to estimate the 3D scene flow in step 403. t represents the time sequence number when the point cloud was captured. The 3D scene flow estimation network will be explained in detail later.

[0041] The estimated scene flow is used in step 404. The non-rigid ICP unit 303 receives the estimated scene flow and the point cloud t. In step 404, the estimated scene flow is used to pre-warp the point cloud t.

[0042] In step 405, a non-rigid ICP algorithm is applied to the pre-distorted point cloud t to align it to the canonical model. Standard non-rigid ICP criteria (e.g., point-to-point, point-to-surface, and as rigid as possible) are used to estimate the deformation parameters of the point cloud t.

[0043] The estimated deformation parameters are used to warp the pre-warped point cloud t, and then the warped point cloud t is input into the point cloud fusion unit 304. In step 406, the warped point cloud is merged into the canonical model.

[0044] In step 407, the reconstruction system checks whether the input point cloud is the last frame (the last point cloud input). If the point cloud is not the last point cloud input, steps 400 to 406 are repeated.

[0045] After fusing the last point cloud input, a loop closure process is performed in step 408 to reduce error accumulation. Based on the representation of the reconstructed model, the loop closure process can be implemented using any type of algorithm (e.g., see “PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space” by CRQi, L.Yi, H.Su, and LJ Guibas at the 2017 International Conference on Neural Information Processing Systems in Long Beach, California, and “RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds” by Q.Hu, B.Yang, L.Xie, S.Rosa, Y.Guo, Z.Wang, N.Trigoni, and A.Markham at the 2020 IEEE Conference on Computer Vision and Pattern Recognition in Seattle, Washington).

[0046] After performing the loop closure, a 3D model is extracted from the fused point cloud in step 409. Finally, the extracted 3D model is stored in storage device 206.

[0047] In one embodiment of the invention, a pair of point clouds (i.e., consecutive frames) are input into a neural network to estimate the 3D scene flow for each point. The user evaluates the quality of the 3D scene flow by making the color-decoded 3D scene flow visible on a 2D screen. The system checks whether the estimated 3D scene flow is smooth. If the estimated 3D flow is smooth enough, a non-rigid 3D point cloud fusion process is performed to reconstruct the 3D model of the target object. The following describes... Figure 5 and Figure 6 Describe a 3D scene flow estimation network, and combine it with Figure 7 and Figure 8 Describe the process of adjusting the scene flow estimation parameters.

[0048] Figure 5 The overall architecture of the 3D scene flow estimation network is shown. Specifically, in the 3D scene flow estimation network... Figure 6 Steps 606 and 607 in the algorithm use an as-rigid-as-possible (ARAP) regularization term. Existing scene flow algorithms are not suitable for 3D object reconstruction. By adding an ARAP regularization term to the scene flow estimation network, the algorithm can correctly handle surface deformation. This is an important factor in the reconstruction of non-rigid 3D objects.

[0049] A neural network model is trained in a supervised learning manner to learn 3D scene flow by processing surface deformation using a paired point cloud dataset. The dataset comprises a collection of dynamic object data, including sequenced point clouds of objects. The neural network is trained to explicitly infer surface deformation of point cloud pairs using as-rigid-as-possible (ARAP) constraints. The 3D scene flow estimation network can be used to efficiently estimate correspondences between two point clouds. One embodiment is non-rigid 3D object reconstruction. The estimated 3D scene flow is used to find corresponding point pairs between point clouds. These correspondences are used in the non-rigid point cloud fusion process.

[0050] like Figure 5As shown, the network includes a point feature pyramid and a coarse-to-fine scene flow estimator. In one embodiment of the invention, firstly, two point clouds 500 and 501 (e.g., point clouds t–1 and t from step 403 above) are input into the network. Then, point feature pyramids (PC1-L1, PC1-L2, PC1-L3, PC2-L1, PC2-L2, and PC2-L3) are constructed using point cloud convolutional kernels (PC Conv-L1 502, PC Conv-L2 503, and PC Conv-L3 504). In this process, point cloud convolutional layers of the same level use the same weights (same weights for the same level). Point cloud convolutional kernels can be implemented using any type of algorithm, such as PointNet++ (e.g., see “PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space” presented by CRQi, L.Yi, H.Su, and LJ Guibas at the 2017 International Conference on Neural Information Processing Systems in Long Beach, California) and RandLA-Net (e.g., see “RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point” presented by Q.Hu, B.Yang, L.Xie, S.Rosa, Y.Guo, Z.Wang, N.Trigoni, and A. Markham at the 2020 IEEE Conference on Computer Vision and Pattern Recognition in Seattle, Washington). RandLA-Net ("Efficient Semantic Segmentation for Large-Scale Point Clouds") and KPConv (e.g., see "KPConv: Flexible and Deformable Convolution for Point Clouds" presented by H. Thomas, CRQi, J.-E. Deschaud, B. Marcotegui, F. Goulette, and LJ Guibas at the International Conference on Computer Vision in Seoul, South Korea, 2019). Scene flow estimators 505, 506, and 507 receive point features from the corresponding layers.Additionally, each scene flow estimator receives the estimated scene flow from the estimator below it. Note that the lowest-level estimator receives the initial flow 508. The initial flow can be set to 0. The highest-level estimator outputs the final result of the estimated scene flow 509.

[0051] Figure 6 The details of the scene flow estimator for estimator-L1 505 are shown. Note that the other estimators (estimator-L2 506 and estimator-L3 507) have the same architecture as estimator-L1 505. Estimator-L1 receives point features PC1-L1 600 and PC2-L1 601 as input. Additionally, estimator-L1 also receives flow-L1 602 from the next layer estimator-L2 506. First, an initial scene flow for each point feature PC1-L1 600 and PC2-L2 601 is computed by linearly mixing (603) the flow-L1 602. After the linear mixing process, an upsampled scene flow (604) is obtained. The warping layer (605) uses the initial scene flow to warp the point features. After the distortion, in step 606, the 3D cost is calculated based on the learned point-to-point cost, point-to-area cost, as-rigid-as-possible (ARAP) cost, and point-to-point distance cost in the feature space. The ARAP cost for each point i is calculated. First, the K nearest neighbors of point i are searched in the input point cloud. For each nearest neighbor j (j = 1, 2, ..., K), the following ARAP energy is calculated:

[0052] ARAP energy: ||(t) i -t j )-R i (t′ i -t′ j )|| 2

[0053] Among them, t i and t j These are the translation parameters for points i and j, respectively, R i This is the 3D rotation matrix of point i. The ARAP cost of point i is calculated using K ARAP energies. Finally, the neural network refines the initial 3D scene flow in scene flow refiner 607 based on the 3D cost. To refine the scene flow, point cloud convolution kernels are used in the neural network.

[0054] Figure 7 This is a flowchart of the scene flow estimation parameter adjustment process. This process is optional for non-rigid 3D reconstruction. (If omitted...) Figure 7The process shown does not require user interaction. During this process, the user can adjust the frame skipping parameters (i.e., 3D scene flow estimation parameters) used for 3D object reconstruction to control the stability, efficiency, and accuracy of the reconstruction process. First, in step 700, the 3D scene flow is estimated in the same manner as in step 403. In step 701, the estimated 3D scene flow is projected onto a 2D image plane and displayed on display 207. For intuitive visualization, the 3D scene flow is shaded and / or rendered in 3D space. The user can qualitatively check the estimation results in step 702. If the estimated 3D scene flow is sufficiently smooth, the user ends the parameter adjustment process. If the estimated 3D scene flow is not smooth enough, the user changes the frame skipping parameters in step 703. In step 403 above, consecutive frames (i.e., point clouds t–1 and t) are used for 3D scene flow estimation. By adjusting the frame skipping parameters, the interval between two point clouds can be changed to t–3 and t, t–5 and t, and so on. The process returns to step 700 to re-estimate the 3D scene flow using the adjusted parameters, and in step 701, the user can inspect the estimated scene flow.

[0055] Figure 8 An example of a user interface for adjusting 3D scene flow estimation parameters is shown. User 802 can adjust the frame skipping parameters by sliding the slider 801 at the bottom of the screen 800 through a touch interface (user interface 208 on display 207). The visualization results from step 701 are displayed on display 207. The user can interactively check the quality of the 3D scene flow estimation results before and after parameter adjustment. Although this example uses a smartphone, the user interface can be implemented using display devices and pointing devices (such as mouse devices).

[0056] In another embodiment, the 3D scene flow estimation generated by existing algorithms (e.g., refer to US Patent Application No. US20190057509A1, filed August 16, 2017, entitled “Learning rigidity of dynamic scenes for three-dimensional scene flow estimation”, by Z.Lv, K.Kim, D.Sun, AJTroccoli, and J.Kautz, and “FlowNet3D: Learning Scene Flow in 3D Point Clouds”, published by X.Liu, CRQi, and LJGuibas at the IEEE Conference on Computer Vision and Pattern Recognition in Long Beach, California in 2019) can be refined by regularization terms and optimization.

[0057] According to embodiments of the present invention, even if the reconstructed target exhibits significant motion, non-rigid objects can be accurately and stably modeled. Furthermore, since the algorithm does not use any prior models, embodiments of the present invention can be applied to various types of non-rigid objects.

[0058] The above description merely discloses exemplary embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Those skilled in the art will understand that all or part of the processes for implementing the above embodiments, as well as equivalent modifications made according to the claims of the present invention, should fall within the scope of the present invention.

Claims

1. A device for reconstructing non-rigid objects, characterized in that, The device includes: Depth sensor; A processor is used to repeatedly execute the following steps: A depth sensor is used to capture point clouds, where the first point cloud is set as a canonical model; The captured point cloud and the previous point cloud are used to estimate the scene flow; The estimated scene flow is used to pre-distort the captured point cloud; The deformation parameters of the captured point cloud are estimated using the non-rigid iterative nearest point (ICP) criterion. The deformation parameters are used to distort the pre-distorted point cloud; The distorted point cloud is merged into the canonical model.

2. The device according to claim 1, characterized in that, The device also includes a display, and the estimated scene flow includes repeating the following steps before receiving user input to end the following steps: The captured point cloud and the previous point cloud are used to estimate the scene flow; The estimated scene flow is projected onto the plane and displayed on the display. Receive user input to adjust scene flow estimation parameters.

3. The device according to claim 1 or 2, characterized in that, The estimated scene stream includes: The first point feature is generated from the previous point cloud using the first point cloud convolution kernel, and the first point feature is input into the next layer of the first point cloud convolution kernel. A second point feature is generated from the captured point cloud using a second point cloud convolution kernel, and the second point feature is input into the next layer of the second point cloud convolution kernel; The scene flow is estimated using the first point feature, the second point feature, and the scene flow estimated in the next layer.

4. The device according to claim 3, characterized in that, The step of estimating the scene flow using the first point feature, the second point feature, and the scene flow estimated in the next layer includes: The point features are distorted using an initial scene flow calculated based on the scene flow estimated in the next layer; Calculate the 3D cost based on the most rigid possible cost; The initial scene flow is refined based on the 3D cost to obtain the estimated scene flow.

5. A method for reconstructing a non-rigid object, characterized in that, The method includes repeatedly performing the following steps: A depth sensor is used to capture point clouds, where the first point cloud is set as a canonical model; The captured point cloud and the previous point cloud are used to estimate the scene flow; The estimated scene flow is used to pre-distort the captured point cloud; The deformation parameters of the captured point cloud are estimated using the non-rigid iterative nearest point (ICP) criterion. The deformation parameters are used to distort the pre-distorted point cloud; The distorted point cloud is merged into the canonical model.

6. A stored program storage medium, characterized in that, When the program is executed by a processor, the program causes the processor to perform the method according to claim 5.

Images