A method of registration of a non-rigid shape and storage medium

Abstract

The application discloses a non-rigid shape registration method and a storage medium, and comprises the following steps: S1: inputting a set of shapes for training into a deep function mapping network to obtain a first function mapping relationship with frequency domain consistency; S2: obtaining features of the set of shapes for training, and obtaining a second function mapping relationship with space domain consistency based on cycle consistency; S3: calculating a loss function according to the first function mapping relationship and the second function mapping relationship, and training the deep function mapping network based on the loss function; and S4: inputting a set of shapes to be registered into the deep function mapping network trained in step S3 to obtain a function mapping relationship of the set of shapes to be registered. The non-rigid shape registration method and the storage medium improve the generalization ability of the model while ensuring the accuracy of non-rigid shape registration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a non-rigid shape registration method and a storage medium. BACKGROUND

[0002] Non-rigid shape registration is a basic task in shape analysis, and plays a key role in application scenarios including texture transfer, voxel parameterization and shape analysis. At present, in the non-rigid shape registration method based on deep learning, the deep function mapping has been widely used. As a kind of spectral mapping representation, it can encode the mapping relationship between two shapes into a matrix, and represent the mapping prior condition between two shapes in a simple algebraic form. Through the method of deep learning, the characteristics of the function mapping prior condition are optimized, and many unsupervised methods can achieve the same accuracy as supervised methods. However, the existing methods mostly have the shortcomings of data overfitting and poor generalization ability, and in addition, when the shape is obviously distorted or distorted, the matching accuracy of some methods will be seriously reduced.

[0003] The disclosure of the above background art content is only used to assist in understanding the concept and technical solution of the present application, and it does not necessarily belong to the prior art of the present patent application. In the absence of explicit evidence that the above content has been disclosed on the filing date of the present patent application, the above background art should not be used to evaluate the novelty and inventiveness of the present application. SUMMARY

[0004] To solve the above technical problems, the present application provides a non-rigid shape registration method and a storage medium, which can ensure the accuracy of non-rigid shape registration while improving the generalization ability of the model.

[0005] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0006] In a first aspect, the present application discloses a non-rigid shape registration method, comprising the following steps:

[0007] S1: input a group of shapes for training into a deep function mapping network to obtain a first function mapping relationship with frequency domain consistency;

[0008] S2: obtain the features of the group of shapes for training, and obtain a second function mapping relationship with spatial domain consistency based on cycle consistency;

[0009] S3: calculate a loss function according to the first function mapping relationship and the second function mapping relationship, and train the deep function mapping network based on the loss function;

[0010] S4: inputting a set of shapes to be registered into the deep function mapping network trained in step S3 to obtain a function mapping relationship of the set of shapes to be registered.

[0011] Preferably, step S1 specifically comprises:

[0012] S11: generating shapes for training based on the deep function mapping network 、 corresponding high-dimensional representation matrices 、 , and extracting shapes 、 corresponding feature matrices 、 ;

[0013] S12: calculating shapes 、 corresponding smooth feature matrices 、 according to the high-dimensional representation matrices 、 feature matrices 、 , wherein , , ;

[0014] S13: calculating an optimal frequency domain mapping function 、 based on the smooth feature matrices :

[0015]

[0016] In the formula, is a first function mapping relationship between shapes 、 , , is a regularization term used to ensure the structural characteristics of the first function mapping relationship

[0017] S14: obtaining the first function mapping relationship with frequency domain consistency by taking tends to 0 as the target. .

[0018] Preferably, step S2 specifically comprises:

[0019] S21: obtaining shapes 、 corresponding high-dimensional representation matrices 、 , and smooth feature matrices​ 、 ;

[0020] S22: shape in the row and shape in the row :

[0021]

[0022] wherein, denotes the row the row, , , 、 respectively the number of shapes , shape ;

[0023] S23: calculate the residual between shape in the row and shape in the row

[0024]

[0025] wherein, , is a variable;

[0026] S24: calculate the second function mapping relationship between shape , :

[0027]

[0028] wherein, .

[0029] Preferably, in step S3, when training the deep function mapping network based on the loss function, the value of the variable increases gradually with the increase of the iteration rounds of the network.

[0030] Preferably, the high-dimensional representation matrix , and the smooth feature matrix , in step S21 are obtained from the step of inputting a group of shapes for training into the deep function mapping network in step S1.​

[0031] Preferably, step S3 specifically comprises: calculating a first function mapping relationship and a second function mapping relationship based on the MSE loss function : , and training the deep function mapping network based on the MSE loss function .

[0032] In a second aspect, the present application discloses a storage medium, wherein the storage medium stores a computer program, and the computer program is arranged to be run by a processor to execute the steps of the non-rigid shape registration method of the first aspect.

[0033] Compared with the prior art, the present application has the beneficial effects that: the non-rigid shape registration method proposed by the present application is based on cycle consistency and uses an unsupervised learning framework, so that the deep function mapping network learned by the network can maintain consistency in both the spatial domain and the frequency domain; on the one hand, under the restriction condition of ensuring spatial domain consistency and frequency domain consistency, the present application can effectively alleviate the problem of data overfitting of existing methods, has good generalization ability, and in the problem of non-rigid shape registration, the accuracy of the present application is significantly better than that of existing methods; on the other hand, the present application is based on an unsupervised learning method, so it does not need to perform complex pre-processing on the input data, and can more reasonably process the registration problem between shapes with severe distortion; compared with existing methods, the present application can achieve better accuracy under more universal experimental settings. In addition, the present application calculates the loss function based on the obtained first function mapping relationship and second function mapping relationship, and trains the deep function mapping network based on the loss function, and the overall uses a double-branch network structure, which can be applied to various deep function mapping networks of the prior art, effectively improving the performance of the original network.

[0034] In a further scheme, a variable is added in the calculation of the second function mapping relationship, through a normalized exponential function, different weights are given to the values in the transformation matrix, an optimized second function mapping relationship with spatial domain consistency is realized, and it is ensured that the conversion process is derivable, thereby further optimizing the consistency of the deep function mapping between the spatial domain and the frequency domain. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a non-rigid shape registration method flowchart of the preferred embodiment of the present application;

[0036] Figure 2 is a non-rigid shape registration flowchart based on spatial domain and frequency domain consistency proposed by the present application;

[0037] Figure 3 is a single-branch network structure, and the frequency consistency and the spatial consistency of the double-branch network structure proposed by the present application under different iteration rounds are shown in the following diagrams:

[0038] Figure 4 is a comparison diagram of the shape registration visualization results of the method of the present application and the method of the prior art for shapes with severe distortion;

[0039] Figure 5 is a comparison diagram of the paired results of the method of the present application and the method of the prior art based on the FAUST_r pre-training model for optimization on two groups of animal data. DETAILED DESCRIPTION

[0040] The embodiments of the present application are described in detail below. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present application and its applications.

[0041] It should be noted that when an element is referred to as being "fixed" or "set" on another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, the connection can be for fixing or for circuit / signal communication.

[0042] It should be understood that the terms "length", "width", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the embodiments of the present application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0043] In addition, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0044] Deep Functional Maps (DFM) is a deep learning-based non-rigid shape matching technique that aligns points between two shapes. DFM uses functional maps to describe the relationship between shapes, which is a method of mapping each point of one shape to the corresponding point on another shape. DFM automatically learns the functional map through a deep neural network, thereby achieving shape alignment. The advantage of DFM is that it can handle non-linear deformation between different shapes, and can handle missing data in shapes. In addition, DFM can also be applied to shape generation and editing, etc. The basic idea of DFM is to represent shapes as functions, and the matching between functions can be converted into a vector matching problem in function space. In this process, DFM uses deep learning technology to learn a functional mapping network that maps the function of one shape to the function of another shape, thereby achieving alignment between shapes. However, the main challenge of existing DFM is how to design a robust and effective functional mapping network to deal with complex deformation between different shapes.

[0045] The inventors found through research that the prior art mainly has the following shortcomings: first, only single consistency in the frequency domain or the spatial domain is concerned, and the two are not effectively linked together; second, data overfitting and poor generalization ability are common; third, in the process of non-rigid shape matching with severe distortion, there is a lack of accuracy. In view of these shortcomings, the inventors found through research that under certain conditions, the function mapping relationship learned by the model is consistent in the frequency domain, but this does not guarantee that the mapping function is consistent in the spatial domain. Based on this, the inventors propose a dual-branch network structure that uses unsupervised learning to use cyclic consistency to make the function mapping relationship learned by the model consistent in both the frequency domain and the spatial domain, effectively solving the data overfitting problem existing in the prior art, having strong model generalization ability, and effectively handling non-rigid shape registration problems with obvious distortion, and having obvious improvement in accuracy compared to existing methods.

[0046] The preferred embodiment of the present application uses an unsupervised learning method, and adds cyclic consistency in the calculation process of the loss function. First, the frequency domain network output is converted to spatial domain output through a transformation matrix, and then the mean square error of the two outputs is calculated. By optimizing the improved loss function, the network model is optimized, so that the function mapping relationship learned by the model can be consistent in both the frequency domain and the spatial domain.

[0047] As shown in Figure 1 The preferred embodiment of the present application discloses a non-rigid shape registration method, comprising the following steps:

[0048] S1: input a set of shapes for training to a deep function mapping network to obtain a first function mapping relationship with frequency domain consistency;

[0049] The deep function mapping network in the preferred embodiment of the present application adopts a deep function mapping network known in the prior art, the specific structure and parameters of which are not the innovation of the present application, and the deep function mapping network can be selected in different types of neural network structures and parameters, including but not limited to using different feature encoding methods, activation functions, etc., which are not limited herein, and any deep function mapping network that can obtain a function mapping relationship can be used.

[0050] The step S1 specifically comprises:

[0051] S11: generating shapes for training based on the deep function mapping network , corresponding high-dimensional representation matrices , and extracting shapes , corresponding feature matrices , . Figure 2 in the deep function mapping network refers to the parameters of the deep function mapping network.

[0052] S12: calculating shapes , corresponding smooth feature matrices , based on the high-dimensional representation matrices , and the feature matrices , , wherein , , .

[0053] S13: calculating an optimal frequency domain mapping function , based on the smooth feature matrices :

[0054]

[0055] In the formula, is a first function mapping relationship between shapes , , , is a regularization term used to ensure the structural characteristics of the first function mapping relationship .

[0056] S14: taking​ tends to 0 as the target, to obtain the first function mapping relationship with frequency domain consistency .

[0057] In combination Figure 2 with the double-branch network drawn in the present application, the double-branch network referred to in the present application is composed of two branch networks that obtain the first function mapping relationship and the second function mapping relationship respectively. Specifically, in the present step, based on the existing deep function mapping network, the network input is a set of three-dimensional shapes in the form of triangular patches, denoted as , , the shape has points, and the shape has points. For each shape, the first k eigenvectors are calculated by the Laplace-Beltrami operator to generate a high-dimensional representation matrix of each shape , and their pseudo-inverse , . Then, a feature extractor is used to calculate the feature matrix corresponding to each shape , , based on and , it is mapped to a high-dimensional frequency domain representation to obtain a set of associated smooth feature matrices , . Through the associated smooth feature matrix and the regularization layer , the optimal frequency domain mapping function , is calculated. The definition of

[0058]

[0059]

[0060] wherein is a regularization term used to ensure the structural characteristics of (the first function mapping relationship between the shapes , ). Let , the function mapping relationship between the two shapes is continuously optimized so that tends to 0, thereby obtaining the first function mapping relationship with frequency domain consistency .

[0061] S2: Obtain the features of a set of shapes for training, based on the cyclic consistency, to obtain the second function mapping relationship with spatial domain consistency;

[0062] In the step S2, the following steps are included:

[0063] S21: Obtain shapes for training 、 corresponding high-dimensional representation matrix 、 and smooth feature matrix 、 ;

[0064] The high-dimensional representation matrix 、 and smooth feature matrix 、 in this step are obtained from the step of inputting a group of shapes for training into the deep function mapping network in step S1.

[0065] S22: Based on the high-dimensional representation matrix 、 and smooth feature matrix 、 , the residual error between the i-th row of shape and the j-th row of shape

[0066]

[0067] In the formula, represents the i-th row of the j-th column of 、 are respectively the number of points of shape

[0068] S23: the residual error between the i-th row of shape and the j-th row of shape

[0069]

[0070] ​​​​​​​​​​​​​​​​​​​​In the formula, , is a variable;

[0071] S24: Calculate the second function mapping relationship between the shapes , :

[0072]

[0073] In the formula, .

[0074] Specifically, in this step, the cyclic consistency is utilized, when , for a group of shapes , , the frequency domain mapping function is cyclically consistent in the empty space function mapping formed by the columns of . For each shape, the function mapping relationship in the frequency domain can be converted into the corresponding empty space mapping relationship through the pre-computed smooth feature matrix , and the nearest neighbor search. According to the point number , of the two shapes, a group of coordinate columns and are formed, and the residual between the th row of the shape and the th row of the shape is calculated:

[0075]

[0076] wherein, represents the th row of , represents the th row of . Thus, a point-by-point corresponding function (the th row of the shape and the th row of the shape ) can be calculated:

[0077]

[0078] wherein, , is a variable;

[0079] ​​In order to be able to train the consistency of the frequency domain and the spatial domain mapping function through the network, an optimized spatial domain function mapping is adopted in this step, a variable a is added, different weights are given to the values in the transformation matrix through the normalized exponential function, an optimized spatial domain function mapping is realized, and it is ensured that the conversion process is derivable. Finally, the second function mapping with spatial domain consistency can be calculated . .

[0080] S3: Calculate the loss function according to the first function mapping relationship and the second function mapping relationship, and train the deep function mapping network based on the loss function;

[0081] In this step, the MSE loss is used to ensure the consistency of the frequency domain and the spatial domain: . The deep function mapping network is trained through the MSE loss function , wherein, in the process of network training, the initial value of the variable a is a small number, which helps the network to basically unify the function mapping relationship between the two shapes in the frequency domain and the spatial domain at the initial stage of training. With the increase of the iteration rounds of the network, the value of the variable a will gradually increase, requiring the network to match more accurately, effectively solving the problem that the variable a is too small to effectively unify the consistency of the frequency domain and the spatial domain or the problem that the variable a is too large to match accurately at the early stage of the network.

[0082] Wherein, the deep function mapping network of this step can be the same as the deep function mapping network in step S1.

[0083] S4: Input the group of shapes to be registered into the deep function mapping network trained in step S3 to obtain the function mapping relationship of the group of shapes to be registered.

[0084] Based on the idea of cyclic consistency, this embodiment adopts a simple and effective double-branch unsupervised learning framework, so that the deep function mapping relationship learned by the network has consistency in the frequency domain and also has consistency in the spatial domain, effectively improving the generalization ability and the ability to handle complex distortion of the model. In addition, the double-branch network proposed in this embodiment can be applied to the existing deep function mapping network to improve the performance of the original network.

[0085] As Figure 3The figure shows the spatial consistency and the frequency consistency of the single-branch network structure (i.e. the deep function mapping network adopted alone) and the double-branch network proposed in the embodiment of the present application under different iteration rounds, wherein the line A represents the frequency consistency convergence result of the single-branch network, the line B represents the frequency consistency convergence result of the double-branch network, the line C represents the spatial consistency convergence result of the single-branch network, and the line D represents the spatial consistency convergence result of the double-branch network. As can be seen from the figure, the single-branch network cannot effectively converge in spatial consistency with the increase of the network training rounds, but the double-branch network can effectively converge in either the frequency consistency or the spatial consistency.

[0086] The non-rigid shape registration method proposed in the embodiments of the present application can effectively alleviate the problem of data overfitting and has strong generalization ability, can well handle the registration problem of severe deformation, and has obvious improvement in accuracy compared with existing algorithms. Specifically, it is verified through experiments that on the six data sets of FAUST_r, SCAPE_r, SHREC19_r, DT4D-H, SMAL_r and TOSCA_r, the method of non-rigid shape registration proposed in the embodiments of the present application can obtain good accuracy by using different experimental training configurations. Among them, FAUST_r, SCAPE_r and SHREC19_r are three human body based data sets, DT4D-H is a human body like data set containing severe deformation, and SMAL_r and TOSCA_r are two animal based data sets. In order to verify the generalization ability of the present application, first, the network is trained on the training set of FAUST_r or SCAPE_r, and then the registration test is performed on the test set of FAUST_r, SCAPE_r and SHREC19_r, and the matching accuracy is judged by the geodesic error. Compared with the existing optimal algorithm, the embodiments of the present application reduce the error rate by 41% on the SHREC19_r data set. For animal data sets, first, the network is trained on the SMAL_r training set and tested on the SMAL_r and TOSCA_r test sets, and compared with the prior art method, the smallest average geodesic error is obtained. In order to test the ability of the present application to handle severe distortion, first, the network is trained on the training set of DT4D-H data set and tested on the test set of DT4D-H, FAUST_r and SCAPE_r, and the performance of the present application on the three data sets is better than the existing algorithm, which shows that the present method can well handle the distorted shape and learn effective features from it, and generalize it to the regular human body data set.

[0087] Figure 4is a schematic diagram of the results of shape registration visualization of the method of the present application and the prior art method with severely distorted shapes; wherein the network is trained on the DT4D-H training set and tested on the DT4D-H test set, the first row (Ours) from bottom to top is the registration result of the present application, the second row (Att. FMaps) and the third row (UDMSM) are the registration results of the currently optimal algorithm. The figure is a visualization of the registration accuracy between the original shape and the target shape, and the Source in the figure represents the original shape, and the three shapes on the right are the target shapes. Map the original shape to the target shape, if the color on the target shape is more consistent with the color on the original shape, the registration effect is better, so from the figure it can be seen that in the same set of original shape to target shape registration process, the accuracy of the present application is much better than other methods.

[0088] Figure 5 is a schematic diagram of the registration results of the method of the present application and the prior art method based on the FAUST_r pre-trained model on two sets of animal data. The second column (Ours) is the registration result of the present application, the third column (Att. FMaps) and the fourth column (UDMSM) are the registration results of the currently optimal algorithm. The figure is also a visualization of the registration accuracy between the original shape and the target shape, and the Source in the figure represents the original shape. In this figure, the texture is added to better show the registration result of the present application. When the texture of the original shape and the target shape is completely consistent, it means that the registration accuracy is high, and if the texture is distorted or blurred, it means that there is a large error between the two shapes. Therefore, it can be seen that the registration result of the present application is significantly better than the remaining algorithms.

[0089] Among them, the non-rigid shape registration method proposed by the present application can also be applied to more extensive fields, such as replacing the input of the network with incomplete shapes or point cloud data with noise, etc. In addition, the non-rigid shape registration method proposed by the present application is also very beneficial to shape registration, shape deformation processing and other operations.

[0090] In the registration process of two shapes, cycle consistency has always been regarded as a strong prior for optimizing the mapping function, but the inventors found through research that under certain conditions, the function mapping relationship learned by the model is already consistent in the frequency domain, but this cannot guarantee that the mapping function is consistent in the spatial domain. Based on this finding, the preferred embodiment of the present application proposes a brand new deep function mapping network based on unsupervised learning, which uses cycle consistency to make the function mapping relationship learned by the model not only consistent in the frequency domain, but also harmonious and consistent in the spatial domain. Through this way of estimating function mapping in the frequency domain and the spatial domain respectively, the overfitting problem in network training is effectively alleviated, and the generalization ability and the ability to handle distorted shapes of the model are improved.

[0091] Another preferred embodiment of the present application discloses a storage medium, wherein the storage medium stores a computer program, and the computer program is arranged to be run by a processor to execute the steps of the non-rigid shape registration method described in the above preferred embodiment. Optionally, the above storage medium can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0092] The background section of the present application can contain background information about the problems or environment of the present application, rather than the prior art described by others. Therefore, the content contained in the background section is not an admission by the applicant of prior art.

[0093] The above description is further to the present application in conjunction with specific / preferred embodiments, and cannot be deemed as limiting the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, they can make several substitutions or variations to the described embodiments, and these substitutions or variations shall be deemed as falling within the protection scope of the present application. In the description of the present application, the description of the terms "an embodiment", "some embodiments", "a preferred embodiment", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are contained in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. Furthermore, the skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction. Although the embodiments of the present application and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope defined by the appended claims.

Claims

1. A registration method for non-rigid shapes, characterized in that, Includes the following steps: S1: Input a set of shapes for training into a deep function mapping network to obtain a first function mapping relationship with frequency domain consistency; S2: Obtain the features of the set of shapes used for training, and based on cycle consistency, obtain a second function mapping relationship with spatial domain consistency; S3: Calculate the loss function based on the first function mapping relationship and the second function mapping relationship, and train the deep function mapping network based on the loss function; S4: Input the set of shapes to be registered into the deep function mapping network trained in step S3 to obtain the function mapping relationship of the set of shapes to be registered; Specifically, step S1 includes: S11: Generates shapes for training based on a deep function mapping network. , The corresponding high-dimensional representation matrix , and extract the shape , Corresponding feature matrix , ; S12: Based on the high-dimensional representation matrix , Feature matrix , Calculate shape , The corresponding smooth feature matrix , ,in , , ; S13: Based on smooth feature matrix , Calculate the optimal frequency domain mapping function : In the formula, For shape , The first functional mapping relationship between them , This is used to guarantee the mapping relationship of the first function. Regularization term of structural properties S14: with With the goal of approaching 0, a first function mapping relationship with frequency domain consistency is obtained. ; Specifically, step S2 includes: S21: Obtain the shape for training , The corresponding high-dimensional representation matrix , and smooth feature matrix , ; S22: shape The Middle Rows and shapes The Middle Residual between rows : In the formula, express The OK No. OK, , , , Shapes ,shape Points; S23: Calculate shape The Middle Rows and shapes The Middle Between lines In the formula, , As a variable; S24: Calculate shape , The second function mapping relationship between them : In the formula, .

2. The registration method for non-rigid shapes according to claim 1, characterized in that, In step S3, when training the deep function mapping network based on the loss function, as the number of iterations of the network increases, the variables... The value gradually increases.

3. The registration method for non-rigid shapes according to claim 1, characterized in that, The high-dimensional representation matrix in step S21 , and smooth feature matrix , It is obtained from the step S1 where a set of shapes for training is input into the deep function mapping network.

4. The registration method for non-rigid shapes according to claim 1, characterized in that, Step S3 specifically includes: based on the mapping relationship of the first function Mapping relationship between the second function Calculate the MSE loss function : And based on the MSE loss function Train a deep function mapping network.

5. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to be run by a processor to perform the steps of the registration method for non-rigid shapes as described in any one of claims 1 to 4.

Images