A method and system for non-rigid point cloud registration from part to whole
Through the Bi-NOFNet model and the bidirectional registration module, the proposed partial to overall non-rigid point cloud registration method solves the problem of insufficient point cloud registration accuracy and robustness in liver surgery, and achieves a more efficient non-rigid point cloud registration effect.
Patent Information
- Application Number
- CN202411729776.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-05-13
- Estimated Expiration
- 2044-11-29
AI Technical Summary
The prior art is difficult to effectively solve the problem of partial to overall non-rigid point cloud registration in liver surgery, especially in the presence of noise, outliers and non-rigid deformation, and the registration accuracy and robustness are insufficient.
A partial to overall non-rigid point cloud registration method is proposed. The mask features of the source point cloud and the target point cloud are obtained through the Bi-NOFNet model, and the binary mask is predicted and optimized using the overlap perception module. Then the point cloud features are extracted through the masked feature extraction module, and the forward and reverse displacement vectors are calculated through the bidirectional registration module for point set registration.
Through deep feature learning and the use of bidirectional registration modules, the accuracy and reliability of point cloud registration are significantly improved, and noise, rotation and non-rigid deformation can be handled more effectively, and is suitable for scenarios with different overlap and noise amplitude.
Smart Images
Figure CN119205865B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of computer-aided point cloud registration, and in particular to a method and system for part-to-whole non-rigid point cloud registration. Background Art
[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.
[0003] The liver point set data obtained during surgery usually contains non-negligible noise and some abnormal data values, which seriously affects the accuracy of registration. Secondly, due to the limited field of view of the endoscope during liver surgery, only part of the liver surface is visible, so the intraoperative point cloud and the complete liver preoperative point cloud partially overlap, which means that the number of points in the intraoperative point cloud point set is less than the number of points in the preoperative point cloud point set during the actual registration process, which makes non-rigid registration more difficult. In addition, during liver surgery, the liver is affected by factors such as organ compression, respiratory movement, and pneumoperitoneum, and will undergo large non-rigid deformations, which will also bring great challenges to the registration problem.
[0004] In order to solve the above problems, existing technologies mainly rely on traditional point cloud registration algorithms, such as the non-rigid iterative closest point (NICP) algorithm, Gaussian mixture model-based methods (CPD, BCPD), etc. Although these methods can handle non-rigid point cloud registration problems to a certain extent, they are rarely used in part-to-whole non-rigid point cloud registration problems and have some limitations, such as dependence on initialization, sensitivity to noise and outliers due to the need to artificially select features or design feature operators, slow iterative optimization speed, and accuracy issues in handling part-to-whole non-rigid point cloud registration problems.
[0005] In recent years, with the development of deep learning technology, some point cloud registration methods based on deep learning have been proposed, but the following problems still exist:
[0006] 1) Most existing technologies are aimed at the problem of whole-to-whole (the number of point sets of the intraoperative point cloud and the preoperative point cloud are the same) non-rigid point cloud registration, such as FlowNet3D, Bi-PointFlowNet, MSBRN, FPT and other methods. Although the above methods can also handle the problem of part-to-whole non-rigid point cloud registration, they perform poorly in registration accuracy and robustness.
[0007] 2) There are also some deep learning-based methods suitable for dealing with part-to-whole point cloud registration problems (such as Predator and OMNet), but these methods mainly focus on the rigid registration problem of point clouds, and are difficult to solve when dealing with non-rigid deformations in image-guided surgery.
[0008] 3) In addition, there are some methods suitable for dealing with the problem of non-rigid point cloud registration from part to whole, such as Lepard. However, the registration method based on the correspondence between point clouds represented by Lepard cannot predict the displacement between point clouds in an end-to-end manner. At the same time, the algorithm executes very slowly and it is difficult to meet actual needs. Summary of the invention
[0009] In order to solve the above problems, the present invention proposes a part-to-whole non-rigid point cloud registration method and system, which obtains rotation-invariant features for the liver point set, calculates overlapping masks and bidirectional displacements, and performs registration from the target point cloud point set to the source point cloud point set. By calculating the bidirectional loss error function, the part-to-whole non-rigid point set registration can be completed more accurately, further improving the accuracy and reliability of the registration.
[0010] According to some embodiments, the present disclosure adopts the following technical solutions:
[0011] A method for part-to-whole non-rigid point cloud registration, comprising:
[0012] Get the source point cloud point set and the target point cloud point set;
[0013] The source point cloud point set and the target point cloud point set are input into the Bi-NOFNet model, and the source point cloud mask features and the target point cloud mask features are respectively extracted using the feature extraction module without mask; then, the binary mask is predicted using the source point cloud mask features and the target point cloud mask features through the overlap perception module, and the binary mask is optimized;
[0014] The optimized binary mask is input into the masked feature extraction module to extract the features of the source point cloud point set and the target point cloud point set;
[0015] Based on the point set features of the source point cloud and the point set features of the target point cloud, the forward displacement vector and the reverse displacement vector are calculated through the bidirectional registration module, so that the point set registration is performed according to the forward displacement vector and the reverse displacement vector.
[0016] According to some embodiments, the present disclosure adopts the following technical solutions:
[0017] A part-to-whole non-rigid point cloud registration system, comprising:
[0018] A data acquisition module is used to acquire a source point cloud point set and a target point cloud point set;
[0019] The unmasked feature extraction module is used to input the source point cloud point set and the target point cloud point set into the Bi-NOFNet model, and use the unmasked feature extraction module to extract the source point cloud mask features and the target point cloud mask features respectively; then, through the overlap perception module, the target point cloud mask features are used to predict the binary mask, and the binary mask is optimized;
[0020] The masked feature extraction module is used to input the optimized binary mask into the masked feature extraction module to extract the features of the source point cloud point set and the target point cloud point set;
[0021] The bidirectional registration module is used to calculate the forward displacement vector and the reverse displacement vector based on the source point cloud point set features and the target point cloud point set features through the bidirectional registration module, so as to perform point set registration according to the forward displacement vector and the reverse displacement vector.
[0022] According to some embodiments, the present disclosure adopts the following technical solutions:
[0023] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for part-to-whole non-rigid point cloud registration.
[0024] According to some embodiments, the present disclosure adopts the following technical solutions:
[0025] A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method of part-to-whole non-rigid point cloud registration is implemented.
[0026] According to some embodiments, the present disclosure adopts the following technical solutions:
[0027] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device implements the method for part-to-whole non-rigid point cloud registration.
[0028] Compared with the prior art, the present invention has the following beneficial effects:
[0029] The present invention discloses a method for non-rigid point cloud registration from part to whole. Through the feature extraction module, it is possible to perform deep feature learning on the source point cloud and the target point cloud, and extract robust point cloud features using the multi-layer perceptron model MLP, maximum pooling operation and transformer model. This deep feature learning not only enhances the model's understanding of point cloud data, but also improves the robustness to noise and rotation that may occur during surgery. At the same time, the proposed feature extraction module is divided into two types: using overlapping masks and not using overlapping masks, which improves the applicability and flexibility of the module.
[0030] The present invention discloses a method for partial to whole non-rigid point cloud registration. The proposed bidirectional registration module not only performs registration from the source point cloud point set to the target point cloud point set, but also performs registration from the target point cloud point set to the source point cloud point set. By calculating the bidirectional loss error function, the partial to whole non-rigid point set registration can be completed more accurately, further improving the accuracy and reliability of the registration.
[0031] The present invention discloses a partial-to-whole non-rigid point cloud registration method, which utilizes an overlapping perception module to output an overlapping mask based on mask features extracted from a target point cloud, and calculates the cross entropy loss between the predicted overlapping mask and the true overlapping mask to optimize the predicted overlapping mask, thereby further improving the registration accuracy and reliability.
[0032] A partial-to-whole non-rigid point cloud registration method disclosed in the present invention has significant improvements over the existing most advanced non-rigid registration methods when dealing with different overlap scenarios (i.e., overlap of 12.5%, 25%, 50%) and different noise amplitude scenarios (i.e., the presence of low noise, medium noise and high noise), which provides strong support for its potential application in image-guided liver surgery. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.
[0034] Figure 1 A flow chart of a method for non-rigid point cloud registration from part to whole according to an embodiment of the present disclosure;
[0035] Figure 2 An architecture diagram of a method for non-rigid point cloud registration from part to whole provided in an embodiment of the present disclosure;
[0036] Figure 3 A flow chart of a feature extraction module provided by an embodiment of the present invention;
[0037] Figure 4 A schematic diagram of a feature extraction module provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0038] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.
[0039] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.
[0040] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0041] Example 1
[0042] A method for non-rigid point cloud registration from part to whole disclosed herein comprises the following steps:
[0043] Step 1: Obtain source point cloud point set and target point cloud point set;
[0044] Step 2: Input the source point cloud point set and the target point cloud point set into the Bi-NOFNet model, and use the feature extraction module without mask to extract the source point cloud mask features and the target point cloud mask features respectively; then use the overlap perception module to predict the binary mask using the source point cloud mask features and the target point cloud mask features, and optimize the binary mask;
[0045] Step 3: Input the optimized binary mask into the masked feature extraction module to extract the source point cloud point set features and the target point cloud point set features;
[0046] Step 4: Based on the features of the source point cloud point set and the target point cloud point set, the forward displacement vector and the reverse displacement vector are calculated through the bidirectional registration module, and then the point set is registered according to the forward displacement vector and the reverse displacement vector.
[0047] As an embodiment, a method for non-rigid point cloud registration from part to whole disclosed in the present invention uses a Bi-NOFNet model to perform deep feature learning on the source point cloud and the target point cloud, such as Figure 2 The specific implementation process is as follows:
[0048] Step 1: Get the source point cloud point set And the target point cloud point set , input into the Bi-NOFNet model;
[0049] Preferably, laser scanners, optical tracking probes, etc. can be selected to obtain the source point cloud set of the patient before and during surgery. And the target point cloud point set .
[0050] Step 2: Use the unmasked feature extraction module FE (w / o M) of the Bi-NOFNet model to extract the source point cloud mask features and the target point cloud mask features respectively;
[0051] The unmasked feature extraction module FE (w / o M) includes three parts: a multi-layer perceptron model MLP (3, 64, 64), an MLP (64, 128, 1024), a maximum pooling layer and a transformer model. The MLP model extracts local features of a point cloud point set by dot product calculation. The maximum pooling layer downsamples the local feature quantity to obtain a global feature, and then fuses the global feature with the local feature. The transformer model extracts the fused features by a self-attention mechanism and a cross-attention mechanism, and then splices the obtained feature vectors to obtain source point cloud mask features and target point cloud mask features.
[0052] This embodiment uses a feature extraction module, such as Figure 4 The feature extraction module shown in the figure includes a multi-layer perceptron model (MLP (3, 64, 64), MLP (64, 128, 1024)), a maximum pooling layer and a transformer model. The MLP model is used to effectively extract local geometric features of the point cloud through dot product calculations, which helps to capture detailed information in the point cloud, which is particularly important for complex structures that need to be accurately aligned in surgical point cloud registration. Compared with global features, local features can better reflect subtle changes in the point cloud, thereby providing richer information for subsequent registration processes.
[0053] The maximum pooling layer is used to downsample the feature vector and reduce the spatial dimension of the feature map while retaining the most important feature information;
[0054] The transformer model is used to extract features from the fused features again through the self-attention mechanism and the cross-attention mechanism, which helps to obtain richer and higher-quality point cloud feature information, while making the extracted feature information more robust; these high-quality and robust feature representations provide a solid foundation for the subsequent bidirectional registration module, which helps to achieve more accurate surgical point cloud registration.
[0055] Specifically, Figure 3 As shown, the steps include:
[0056] Step 201: Source point cloud point set And the target point cloud point set They are respectively sent to the feature extraction module FE (w / o M) without mask;
[0057] Step 202: Source point cloud point set And the target point cloud point set Multiply by the first transformation matrix (3×3 matrix) for transformation, and use the MLP (3, 64, 64) model to input the transformed source point cloud point set And the target point cloud point set Extract source point set features and target point set features;
[0058] Among them, the first transformation matrix is a 3×3 matrix, and the dimension of the source point cloud point set is N S ×3, the source point cloud point set is multiplied by the first transformation matrix to obtain the transformed source point cloud point set.
[0059] Step 203: Multiply the extracted source point set features and target point set features by the second transformation matrix (64×64 matrix) to obtain the local features of the source point set. l S And the local features of the target point set l T , and use the MLP (64, 128, 1024) model to extract the local features of the new source point set h S and local features of the target point set h T ;
[0060] Step 204: Extract the local features of the source point set h S and local features of the target point set h T Multiply the unit vector and obtain the global features of the source point set through maximum pooling and copy operations G S and the global features of the target point set G T ; Among them, the unit vector is a vector whose elements are all 1, and the process is h s Multiplying by a unit vector, h s The dimension is N S ×1024, the unit vector dimension is N S ×1.
[0061] Step 205: Based on the existing local features of the source point set l S And the local features of the target point set l T and global features G S ,G T The source point set fusion feature f is obtained by splicing S And the target point set fusion feature f T , f S , f T Multiply the unit vector to obtain the filtered source point set feature vector and target point set feature vector;
[0062] Step 206: Based on the filtered source point set feature vector and target point set feature vector, a transformer model is used to directly learn a more significant point cloud feature representation for the input source point set feature vector and target point set feature vector, and extract the source point set significant feature vector C S and the salient feature vector C of the target point set T ;
[0063] Step 207: Substituting the source point set salient feature vector C S and the salient feature vector C of the target point set T The final source point cloud mask feature F is obtained by splicing with the corresponding source point cloud point set and target point cloud point set respectively. S And the target point cloud mask feature F T .
[0064] Step 3: Using the Overlap Awareness (OA) module, the predicted binary mask is calculated based on the extracted target point cloud mask features, and then the cross entropy loss is calculated based on the real data mask to optimize the predicted binary mask.
[0065] Specifically, the calculation formula for the binary mask is:
[0066]
[0067] in, To predict the i-th element in the overlap mask vector, is the i-th element of the soft mask vector output by the overlap perception module; is a fixed threshold, which is set to 0.5 in the experiment.
[0068] The true overlap mask is defined as,
[0069]
[0070] in, To predict the i-th element in the mask vector, It represents the i-th element in the target point cloud point set.
[0071] The cross entropy loss between the predicted overlap mask and the true overlap mask is calculated as:
[0072]
[0073] in, is the cross entropy loss function, is the ratio of the non-overlapping area between the source point cloud point set and the target point cloud point set, and They represent the predicted overlap mask and the true overlap mask respectively.
[0074] Step 4: Reuse the masked feature extraction module FE (w / M) to extract the source point cloud point set features and the target point cloud point set features based on the optimized binary mask; the specific steps are as follows:
[0075] Step 401: Input the source point cloud set S and the target point cloud set T and the predicted overlap mask , and send them to the FE (w / M) module respectively;
[0076] Step 402: The source point cloud point set And the target point cloud point set Multiply by the first transformation matrix (3×3) for transformation, and use the MLP (3, 64, 64) model to transform the input transformed source point cloud point set And the target point cloud point set Extracting features of a second source point set and features of a second target point set;
[0077] Step 403: Multiply the extracted second source point set features and the second target point set features by the transformation matrix (64×64) to obtain the second source point set local features l S ' And the local features of the second target point set l T ' , and use the MLP (64, 128, 1024) model to extract the new local features of the second source point set h S ' and the local features of the second target point set h T ' ;
[0078] Step 404: Extract the local features of the second source point set h S ' and the local features of the second target point set h T ' Multiply by the predicted overlap mask , and obtain the global features of the second source point set through maximum pooling and copy operations G S' and the second target point set global features G T ' ;
[0079] Step 405: Based on the local features of the second source point set l S ' And the local features of the second target point set l T ' and global features G S ' , G T ' The second source point set fusion feature f is obtained by splicing S ' And the second target point set fusion feature f T ' , f S ' , f T ' Multiplying the predicted overlapping mask to obtain the filtered second source point set feature vector and the second target point set feature vector;
[0080] Step 406: Based on the filtered source point set feature vector and target point set feature vector, a transformer model is used to directly learn a more significant point cloud feature representation for the input second source point set feature vector and the second target point set feature vector, and a significant feature vector C of the second source point set is extracted. S ' and the second target point set salient feature vector C T ' ;
[0081] Step 407: Substitute the significant feature vector C of the second source point set S ' and the second target point set salient feature vector C T ' The final source point cloud point set features are obtained by splicing with the corresponding point cloud point set. And the target point cloud point set features .
[0082] Step 5: Use the bidirectional registration (BR) module to extract the source point cloud point set features based on the feature extraction module with mask And the target point cloud point set features The forward displacement vector and the reverse displacement vector are calculated.
[0083] Specifically, the bidirectional registration module is composed of a multi-layer perceptron MLP (1024, 512, 256, 128, 64, 3), and the feature representation of the source point cloud point set extracted by the masked feature extraction module And the target point cloud point set feature representation A forward displacement vector and a reverse displacement vector are calculated, wherein the forward displacement vector and the reverse displacement vector correspond to the transformation from the source point cloud point set to the target point cloud point set and the transformation from the target point cloud point set to the source point cloud point set, respectively.
[0084] Among them, the source point cloud point set feature representation extracted by the masked feature extraction module And the target point cloud point set feature representation The forward displacement vector and the reverse displacement vector are calculated, including:
[0085] After the input data passes through each layer of MLP, its dimension will change. For example, the dimension of the input data is m×n, MLP (1024, 512, 256, 128, 64, 3). After the input data passes through the first layer of MLP, the dimension becomes m×1024. After passing through the second layer, the dimension becomes m×512. And so on. After the input data passes through the last layer of MLP, the dimension becomes m×3. That is, the m×3 vector is the calculated displacement vector.
[0086] Furthermore, based on the forward displacement vector, the reverse displacement vector and the real displacement vector reconstruction loss, the Bi-NOFNet model is trained for point set registration, including:
[0087] For the Bi-NOFNet model, the forward and reverse L2 loss functions are calculated based on the L2 distance between displacement vectors; the L2 loss function is:
[0088]
[0089]
[0090] Among them, N s is the number of points in the source point cloud, N t is the number of points in the target point cloud point set; is the true positive displacement vector, To predict the forward displacement vector; is the true reverse displacement vector, To predict the reverse displacement vector.
[0091] Experimental simulation
[0092] In order to verify the performance of the registration method proposed in this disclosure, this embodiment selects a traditional registration method CPD and five deep learning-based algorithms FPT, FLOT, NDP, Bi-PointFlowNet, and MSBRN for comparison with the method proposed in this embodiment. The performance of each model is evaluated based on the evaluation indicators, and the experimental results are shown in Table 1.
[0093] The dataset used in the experiment comes from 615 liver point cloud samples in MedShapeNet, of which 551 are for training, 32 for validation and 32 for testing. Three evaluation indicators, namely mean absolute error (MAE), root mean square error (RMSE) and registration recall (RR), are used to judge the quality of the registration results. The three indicators are expressed as:
[0094]
[0095]
[0096] RR=
[0097] in, is the number of points in the source point cloud, is the true positive displacement vector, To predict the forward displacement vector, is the total number of samples in the test set, It is an artificially set threshold.
[0098] The five different loss functions of this embodiment are used to train the Bi-NOFNet model, and these loss functions are:
[0099] 1. Used in Bi-NOFNet, the forward L2 distance loss calculates the average Euclidean distance between the true forward displacement vector and the predicted forward displacement vector, which is:
[0100]
[0101] The reverse L2 distance loss calculates the average Euclidean distance between the true reverse displacement vector and the predicted reverse displacement vector, which is:
[0102]
[0103] 2. The cross entropy loss function, used in Bi-NOFNet, calculates the difference between the predicted overlap mask and the true overlap mask, which is:
[0104]
[0105] 3. Total training loss function: used in Bi-NOFNet, it is a combination of the cross entropy loss function and the bidirectional L2 distance loss function or the bidirectional Chamfer distance loss function, which is:
[0106]
[0107] in, , in a supervised registration process and Actually and , in the weakly supervised registration process and Actually and .
[0108] As can be seen from Table 1, the disclosed method outperforms other comparison methods in the evaluation indicators mean absolute error (MAE), root mean square error (RMSE) and registration recall rate (RR). In terms of the evaluation indicators MAE, RMSE and RR, the average experimental results of Bi-NOFNet implemented in the present disclosure are 11.31%, 15.46% and 4.47% ahead of the best results of other methods, respectively. The comparison results in the above table show that the method proposed in the present disclosure reduces the erroneous correspondence caused by rigid transformation between point sets by introducing feature extraction modules, overlapping perception modules and bidirectional registration modules, improves the robustness to noise and deformation that may occur during surgery, and improves the accuracy of non-rigid registration.
[0109] Table 1 Model performance evaluation
[0110]
[0111] This embodiment extracts features through a feature extraction module, thereby achieving robustness to changes in noise amplitude. Through this method, the accuracy of point cloud registration in image-guided liver surgery can be significantly improved, providing more reliable support for surgical navigation.
[0112] Example 2
[0113] In one embodiment of the present disclosure, a partial to whole non-rigid point cloud registration system is provided, comprising:
[0114] A data acquisition module is used to acquire a source point cloud point set and a target point cloud point set;
[0115] The unmasked feature extraction module is used to input the source point cloud point set and the target point cloud point set into the Bi-NOFNet model, and use the unmasked feature extraction module to extract the source point cloud mask features and the target point cloud mask features respectively; then, through the overlap perception module, the target point cloud mask features are used to predict the binary mask, and the binary mask is optimized;
[0116] The masked feature extraction module is used to input the optimized binary mask into the masked feature extraction module to extract the features of the source point cloud point set and the target point cloud point set;
[0117] The bidirectional registration module is used to calculate the forward displacement vector and the reverse displacement vector based on the source point cloud point set features and the target point cloud point set features through the bidirectional registration module, so as to perform point set registration according to the forward displacement vector and the reverse displacement vector.
[0118] Example 3
[0119] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the method for part-to-whole non-rigid point cloud registration is implemented.
[0120] Example 4
[0121] A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method of part-to-whole non-rigid point cloud registration is implemented.
[0122] Example 5
[0123] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device implements the part-to-whole non-rigid point cloud registration method for liver surgery.
[0124] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0126] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.
Claims
1. A method for part-to-whole non-rigid point cloud registration, characterized in that: include: Get the source point cloud point set and the target point cloud point set; The source point cloud point set and the target point cloud point set are input into the Bi-NOFNet model, and the mask features of the source point cloud and the mask features of the target point cloud are respectively extracted using a feature extraction module without a mask; The source point cloud point set And the target point cloud point set The source point cloud point set and the target point cloud point set are respectively sent to the unmasked feature extraction module FE (w / o M), and the source point cloud point set and the target point cloud point set are respectively multiplied by the first transformation matrix to obtain the transformed source point cloud point set and the target point cloud point set; the MLP (3,64,64) model is used to input the transformed source point cloud point set And the target point cloud point set Extract source point set features and target point set features; The extracted source point set features and target point set features are multiplied by the second transformation matrix to obtain the local features of the source point set. l S And the local features of the target point set l T , use the MLP (64, 128, 1024) model to extract the local features of the new source point set h S and local features of the target point set h T ; The local features of the source point set are extracted h S and local features of the target point set h T Multiply the unit vector and obtain the global features of the source point set through maximum pooling and copy operations G S and the global features of the target point set G T ; Based on the existing local features of the source point set l S And the local features of the target point set l T and global features G S , G T The source point set fusion feature f is obtained by splicing S And the target point set fusion feature f T , f S , f T Multiply the unit vector to obtain the filtered source point set feature vector and target point set feature vector; The filtered source point set feature vector and target point set feature vector are directly input using the transformer model to learn more significant point cloud feature representations, and the source point set significant feature vector C is extracted. S and the salient feature vector C of the target point set T ; The salient feature vector C of the source point set S and the salient feature vector C of the target point set T The final source point cloud mask feature F is obtained by splicing with the corresponding source point cloud point set and target point cloud point set respectively. S And the target point cloud mask feature F T ; Then, through the overlapping perception module, the target point cloud mask features are used to predict the binary mask, and the binary mask is optimized; The optimized binary mask is input into the masked feature extraction module to extract the features of the source point cloud point set and the target point cloud point set; Based on the optimized binary mask, the source point cloud point set features and the target point cloud point set features are extracted respectively; The specific steps are as follows: Input source point cloud point set S and target point cloud point set T and predicted overlap mask , and send them to the masked feature extraction module FE (w / M) module respectively; The source point cloud point set And the target point cloud point set Multiply by the first transformation matrix to transform, and use the MLP (3, 64, 64) model to transform the input source point cloud point set And the target point cloud point set Extracting features of a second source point set and features of a second target point set; The extracted second source point set features and the second target point set features are multiplied by the transformation matrix to obtain the second source point set local features l S ' And the local features of the second target point set l T ' , and use the MLP (64, 128, 1024) model to extract the new local features of the second source point set h S ' and the local features of the second target point set h T ' ; The local features of the second source point set are extracted h S ' and the local features of the second target point set h T ' Multiply by the predicted overlap mask , and obtain the global features of the second source point set through maximum pooling and copy operations G S ' and the second target point set global features G T ' ; Based on the local features of the second source point set l S ' And the local features of the second target point set l T ' and global features G S ' , G T ' The second source point set fusion feature f is obtained by splicing S ' And the second target point set fusion feature f T ' , f S ' , f T ' Multiplying the predicted overlapping mask to obtain the filtered second source point set feature vector and the second target point set feature vector; Based on the filtered source point set feature vector and target point set feature vector, the transformer model is used to directly learn more significant point cloud feature representations for the input second source point set feature vector and the second target point set feature vector, and the significant feature vector C of the second source point set is extracted. S ' and the second target point set salient feature vector C T ' ; The second source point set salient feature vector C S ' and the second target point set salient feature vector C T ' The final source point cloud point set features are obtained by splicing with the corresponding point cloud point set. And the target point cloud point set features ; Based on the point set features of the source point cloud and the point set features of the target point cloud, the forward displacement vector and the reverse displacement vector are calculated through the bidirectional registration module, so that the point set registration is performed according to the forward displacement vector and the reverse displacement vector.
2. A method for part-to-whole non-rigid point cloud registration as claimed in claim 1, characterized in that: The overlap perception module calculates the predicted binary mask based on the extracted source point cloud mask features and target point cloud mask features, and then calculates the cross entropy loss based on the real data mask. The calculation formula of the binary mask is: in, To predict the i-th element in the overlap mask vector, is the i-th element of the soft mask vector output by the overlap perception module, is a fixed threshold that is set.
3. A method for part-to-whole non-rigid point cloud registration as claimed in claim 1, characterized in that: The optimization of the binary mask includes: optimizing the predicted binary mask by calculating the cross entropy loss based on the true overlapping mask. The true overlapping mask is defined as, in, is the i-th element in the true overlap mask vector, It represents the i-th element in the target point cloud point set; the calculation formula of the cross entropy loss between the predicted overlapping mask and the true overlapping mask is: in, is the cross entropy loss function, is the ratio of the non-overlapping area between the source point cloud point set and the target point cloud point set, and They represent the predicted overlap mask and the true overlap mask respectively.
4. The method for part-to-whole non-rigid point cloud registration according to claim 1, characterized in that: The bidirectional registration module is composed of a multi-layer perceptron MLP (1024, 512, 256, 128, 64, 3), and a forward displacement vector and a reverse displacement vector are calculated based on the feature representation of the source point cloud point set and the feature representation of the target point cloud point set, wherein the forward displacement vector and the reverse displacement vector correspond to the transformation from the source point cloud point set to the target point cloud point set and the transformation from the target point cloud point set to the source point cloud point set, respectively.
5. The method for part-to-whole non-rigid point cloud registration according to claim 1, characterized in that: The forward L2 loss function and the reverse L2 loss function are calculated based on the L2 distance between the displacement vectors; the L2 loss function is: Among them, N s is the number of points in the source point cloud, N t is the number of points in the target point cloud point set; is the true positive displacement vector, To predict the forward displacement vector; is the true reverse displacement vector, To predict the reverse displacement vector.
6. A part-to-whole non-rigid point cloud registration system, characterized in that: include: A data acquisition module is used to acquire a source point cloud point set and a target point cloud point set; The unmasked feature extraction module is used to input the source point cloud point set and the target point cloud point set into the Bi-NOFNet model, and use the unmasked feature extraction module to extract the source point cloud mask features and the target point cloud mask features respectively; Then, through the overlap perception module, the target point cloud mask features are used to predict the binary mask, and the binary mask is optimized; The unmasked feature extraction module includes three parts: a multi-layer perceptron model MLP (3, 64, 64), MLP (64, 128, 1024), a maximum pooling layer and a transformer model. The MLP model extracts local features of the point cloud point set by dot product calculation; the maximum pooling layer downsamples the local features to obtain global features, and then fuses the global features and local features. The transformer model extracts the fused features through the self-attention mechanism and the cross-attention mechanism, and then concatenates the obtained feature vectors to obtain the source point cloud mask features and the target point cloud mask features; Multiplying the source point cloud point set and the target point cloud point set by the first transformation matrix respectively to obtain the transformed source point cloud point set and the target point cloud point set; The extracted source point set features and target point set features are multiplied by the second transformation matrix to obtain the local features of the source point set. l S And the local features of the target point set l T , use the MLP (64, 128, 1024) model to extract the local features of the new source point set h S and local features of the target point set h T ; The local features of the source point set are extracted h S and local features of the target point set h T Multiply the unit vector and obtain the global features of the source point set through maximum pooling and copy operations G S and the global features of the target point set G T ; Based on the existing local features of the source point set l S And the local features of the target point set l T and global features G S , G T The source point set fusion feature f is obtained by splicing S And the target point set fusion feature f T , f S , f T Multiply the unit vector to obtain the filtered source point set feature vector and target point set feature vector; The filtered source point set feature vector and target point set feature vector are directly input using the transformer model to learn more significant point cloud feature representations, and the source point set significant feature vector C is extracted. S and the salient feature vector C of the target point set T ; The salient feature vector C of the source point set S and the salient feature vector C of the target point set T The final source point cloud mask feature F is obtained by splicing with the corresponding source point cloud point set and target point cloud point set respectively. S And the target point cloud mask feature F T ; The masked feature extraction module is used to input the optimized binary mask, the source point cloud point set and the target point cloud point set into the masked feature extraction module to extract the features of the source point cloud point set and the features of the target point cloud point set; Based on the optimized binary mask, the source point cloud point set features and the target point cloud point set features are extracted respectively; The specific steps are as follows: Input source point cloud point set S and target point cloud point set T and predicted overlap mask , and send them to the FE (w / M) module respectively; The source point cloud point set And the target point cloud point set Multiply by the first transformation matrix to transform, and use the MLP (3, 64, 64) model to transform the input source point cloud point set And the target point cloud point set Extracting features of a second source point set and features of a second target point set; The extracted second source point set features and the second target point set features are multiplied by the transformation matrix to obtain the second source point set local features l S ' And the local features of the second target point set l T ' , and use the MLP (64, 128, 1024) model to extract the new local features of the second source point set h S ' and the local features of the second target point set h T ' ; The local features of the second source point set are extracted h S ' and the local features of the second target point set h T ' Multiply by the predicted overlap mask , and obtain the global features of the second source point set through maximum pooling and copy operations G S ' and the second target point set global features G T ' ; Based on the local features of the second source point set l S ' And the local features of the second target point set l T ' and global features G S ' , G T ' The second source point set fusion feature f is obtained by splicing S ' And the second target point set fusion feature f T ' , f S ' , f T ' Multiplying the predicted overlapping mask to obtain the filtered second source point set feature vector and the second target point set feature vector; Based on the filtered source point set feature vector and target point set feature vector, the transformer model is used to directly learn more significant point cloud feature representations for the input second source point set feature vector and the second target point set feature vector, and the significant feature vector C of the second source point set is extracted. S ' and the second target point set salient feature vector C T ' ; The second source point set salient feature vector C S ' and the second target point set salient feature vector C T ' The final source point cloud point set features are obtained by splicing with the corresponding point cloud point set. And the target point cloud point set features ; The registration module is used to calculate the forward displacement vector and the reverse displacement vector through the bidirectional registration module based on the source point cloud point set features and the target point cloud point set features, so as to perform point set registration according to the forward displacement vector and the reverse displacement vector.
7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for part-to-whole non-rigid point cloud registration described in any one of claims 1 to 5 is implemented.
8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, a part-to-whole non-rigid point cloud registration method as described in any one of claims 1-5 is implemented.
9. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement a part-to-whole non-rigid point cloud registration method as described in any one of claims 1-5.