Knee joint point cloud registration method, device, equipment and medium

Abstract

The invention discloses a knee joint point cloud registration method, device and equipment and a medium, and relates to the technical field of medical image processing, and the method can be used for carrying out registration on knee joint point clouds which are obtained by CT (Computed Tomography) image reconstruction and are acquired by point cloud acquisition equipment and have different modalities, and is used for guiding a knee joint replacement operation. According to the method, the registration between the point cloud reconstructed by the CT scanning before the knee replacement operation and the point cloud acquired by the point cloud acquisition equipment during the operation can be effectively realized, the special shape information of the point cloud is effectively processed, the difficulty of low overlapping rate of the point cloud is overcome, and accurate information is provided for augmented reality operation navigation.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and particularly to a knee joint point cloud registration method, device, equipment and medium based on the Transformer architecture. Background Art

[0002] Knee joint replacement surgery refers to a surgery in which an artificial joint prosthesis made of artificial materials is implanted into the knee joint according to the knee joint structure and function to achieve the treatment and improvement of knee joint symptoms and functions. During the surgery, doctors need to accurately position according to the knee joint structure under CT images. Therefore, the three-dimensional visualization technology and augmented reality navigation based on CT images have become important research directions, and this technology is of great significance for reducing accidental injuries and improving postoperative effects. At the same time, it can also reduce the burden on doctors, reduce the excessive dependence of doctors on experience, and reduce the difficulty of training doctors.

[0003] In the above augmented reality surgical navigation system, point cloud registration before and during surgery is a key issue. The knee joint bone surface point cloud obtained by CT scan reconstruction before surgery and the knee joint bone surface point cloud obtained by various point cloud acquisition devices during surgery are different in terms of spatial coordinates, resolution, perspective coverage range, etc. The goal of registration is to solve these problems and achieve accurate registration of the two point clouds to provide accurate information for augmented reality surgical navigation. In the development of point cloud registration technology, methods based on deep learning have better performance.

[0004] In the scenario of knee joint bone point cloud registration before and during surgery, due to the special shape information of the knee joint bone point cloud that needs special attention, and the problem of low overlap rate between the knee joint bone point clouds before and during surgery, the existing configuration methods are difficult to handle the characteristics such as low overlap rate and special shape of the knee joint bone point clouds before and during surgery, resulting in poor registration effects.

[0005] Therefore, how to provide a registration method to solve the technical problem that the existing methods are difficult to handle the characteristics such as low overlap rate and special shape of the knee joint bone point clouds before and during surgery, resulting in poor registration effects, is an urgent technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the above problems, the present invention provides a knee joint point cloud registration method, device, equipment and medium for overcoming the above problems or at least partially solving the above problems. It solves the technical problem that the existing methods are difficult to handle the characteristics such as low overlap rate and special shape of the knee joint bone point clouds before and during surgery, resulting in poor registration effects.

[0007] The present invention provides the following solutions:

[0008] A knee joint point cloud registration method, comprising:

[0009] Obtain a source point cloud and a reference point cloud, where the source point cloud includes the surface point cloud of the knee joint bone obtained by preoperative CT scan reconstruction, and the reference point cloud includes the surface point cloud of the knee joint bone obtained by a point cloud acquisition device during surgery; use a point cloud shape topology information generation module to respectively extract and superimpose the shape topology information of the source point cloud and the reference point cloud to generate a source point surface point cloud and a reference surface point cloud containing shape topology information;

[0010] Perform multi-stage downsampling on the source point surface point cloud and the reference surface point cloud respectively, and perform feature extraction on the obtained multi-stage point clouds to generate source point cloud superpoints and corresponding source point cloud dense points, reference point cloud superpoints and corresponding reference point cloud dense points, source point cloud multi-scale features, and reference point cloud multi-scale features; the dense points include the point cloud obtained by the first-stage downsampling, and the superpoints include the point cloud obtained by the last-stage downsampling;

[0011] Input the source point cloud superpoints, the reference point cloud superpoints, the source point cloud superpoint features, and the reference point cloud superpoint features into a superpoint matching module based on the geometric Transformer architecture to obtain the corresponding relationship between the source point cloud superpoints and the reference point cloud superpoints;

[0012] Expand the corresponding relationship between the source point cloud superpoints and the reference point cloud superpoints through the optimal transport method to obtain the corresponding relationship between the source point cloud dense points and the reference point cloud dense points;

[0013] Use weighted singular value decomposition for the corresponding relationship between dense points in each region to obtain the transformation matrix within the region; the obtained all different transformation matrices are globally optimized to obtain a unique target transformation matrix, so as to apply the target transformation matrix to the source point cloud and the reference point cloud to complete the registration.

[0014] Preferably: the point cloud shape topology information generation module includes a feature encoder, a flow predictor, a flow generator, and a point generator; the feature encoder is used to encode the point cloud to obtain a feature vector, the flow generator and the flow predictor are used to jointly generate a label using the point cloud and the feature vector, and the label includes information on which part each point should be assigned to; the point generator is used to generate a surface point cloud containing shape topology information according to the label, the feature vector, and the point cloud.

[0015] Preferably: the superimposition includes superimposing the shape topology information back into the original point cloud in any one form or a combination of forms such as color, label, and mask.

[0016] Preferably: A multi-stage network based on kernel point convolution is used for feature extraction to obtain the multi-scale features of the source point cloud and the multi-scale features of the reference point cloud.

[0017] Preferably: The superpoint matching module includes geometric structure embedding, self-attention, cross-attention, and feature matching; the geometric structure embedding is used to abstract superpoints into special encodings, the self-attention is used to learn the global connections of superpoints in a single point cloud in geometric space and feature space, the cross-attention is used to further learn the global connections between superpoints of the source point cloud and the reference point cloud, so as to generate mixed features to characterize these connections, and the feature matching also calculates a series of alternative corresponding relationships by calculating feature distances and Gaussian correlation matrices, and uses the topk strategy to select the optimal corresponding relationship among them.

[0018] Preferably: The result of the optimal transport method is to output a cost matrix, which is processed by the sinkhorn algorithm to be transformed into a soft assignment matrix, and a confidence matrix is generated by processing the soft assignment matrix according to the confidence level. Using the confidence matrix and the multiple topk strategy, the corresponding relationships between dense points are obtained.

[0019] Preferably: The weighted singular value decomposition includes multiple transformation matrices calculated according to the dense points corresponding to each pair of superpoints, and a global optimization function is used to calculate the transformation matrix that maximizes the inlier rate of all dense points of the registered source point cloud and reference point cloud, which is used as the target transformation matrix for registering the source point cloud and the reference point cloud.

[0020] A knee joint point cloud registration device for performing the above knee joint point cloud registration method, the device includes:

[0021] A point cloud topological shape information generation unit, configured to obtain a source point cloud and a reference point cloud, where the source point cloud includes the knee joint bone surface point cloud obtained by preoperative CT scan reconstruction, and the reference point cloud includes the knee joint bone surface point cloud obtained by a point cloud acquisition device during surgery; using a point cloud shape topological information generation module to respectively perform shape topological information extraction and superposition on the source point cloud and the reference point cloud, and generating a source point surface point cloud and a reference surface point cloud containing shape topological information;

[0022] A feature extraction unit, configured to respectively perform multi-stage downsampling on the source point surface point cloud and the reference surface point cloud, and perform feature extraction on the obtained multi-stage point clouds to generate source point cloud superpoints and corresponding source point cloud dense points, reference point cloud superpoints and corresponding reference point cloud dense points, source point cloud multi-scale features, and reference point cloud multi-scale features; the dense points include the point clouds obtained by the first-stage downsampling, and the superpoints include the point clouds obtained by the last-stage downsampling;

[0023] A superpoint matching unit, which is configured to input the source point cloud superpoints, the reference point cloud superpoints, the source point cloud superpoint features, and the reference point cloud superpoint features into a superpoint matching module based on a geometric Transformer architecture to obtain the correspondence between the source point cloud superpoints and the reference point cloud superpoints;

[0024] A global point matching unit, which is configured to expand the correspondence between the source point cloud superpoints and the reference point cloud superpoints by an optimal transport method to obtain the correspondence between the source point cloud dense points and the reference point cloud dense points;

[0025] A local-to-global registration unit, which is configured to use weighted singular value decomposition for the correspondence between dense points in each region to obtain an intra-region transformation matrix; and perform global optimization on all the obtained different transformation matrices to obtain a unique target transformation matrix, so as to apply the target transformation matrix to the source point cloud and the reference point cloud to complete registration.

[0026] A knee joint point cloud registration device, the device comprising a processor and a memory:

[0027] The memory is configured to store program code and transmit the program code to the processor;

[0028] The processor is configured to execute the above-mentioned knee joint point cloud registration method according to the instructions in the program code.

[0029] A computer-readable storage medium, which is configured to store program code, and the program code is configured to execute the above-mentioned knee joint point cloud registration method.

[0030] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0031] A knee joint point cloud registration method, device, equipment and medium provided by an embodiment of the present application. This method can register different modality knee joint point clouds reconstructed from CT (Computed Tomography) images and collected by a point cloud acquisition device, and is used for guiding knee joint replacement surgery. It can effectively realize the registration between the preoperative CT scan reconstructed point cloud and the intraoperative point cloud collected by the point cloud acquisition device, effectively process the special shape information of the above-mentioned point clouds, overcome the difficulty of low overlap rate existing in the above-mentioned point clouds, and provide accurate information for augmented reality surgical navigation.

[0032] Of course, it is not necessary for any product implementing the present invention to achieve all the above-mentioned advantages simultaneously. Description of the Drawings

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0034] Figure 1 is a flowchart of the knee joint point cloud registration method provided by an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of the point cloud shape topology information generation module provided by an embodiment of the present invention;

[0036] Figure 3 is a schematic diagram of the Transformer-based superpoint matching module provided by an embodiment of the present invention;

[0037] Figure 4 is a schematic diagram of the knee joint point cloud registration device provided by an embodiment of the present invention;

[0038] Figure 5 is a schematic diagram of the knee joint point cloud registration device provided by an embodiment of the present invention. Specific embodiments

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present invention.

[0040] See Figure 1 , a knee joint point cloud registration method provided by an embodiment of the present invention. As Figure 1 shown, the method may include:

[0041] S101: Obtain a source point cloud and a reference point cloud. The source point cloud includes the knee joint bone surface point cloud reconstructed by preoperative CT scanning, and the reference point cloud includes the knee joint bone surface point cloud obtained by a point cloud acquisition device during the operation; Use the point cloud shape topology information generation module to respectively extract and superimpose the shape topology information of the source point cloud and the reference point cloud to generate a source surface point cloud and a reference surface point cloud containing shape topology information;

[0042] In specific implementation, the embodiment of the present application can provide that the point cloud shape topology information generation module includes a feature encoder, a flow predictor, a flow generator, and a point generator; the feature encoder is used to encode the point cloud to obtain a feature vector, and the flow generator and the flow predictor are used to jointly generate a label using the point cloud and the feature vector, where the label includes information on which part each point should be assigned to; the point generator is used to generate a surface point cloud containing shape topology information according to the label, the feature vector, and the point cloud.

[0043] The superposition includes superposing the shape topology information back into the original point cloud in any one form or a combination of several forms such as color, label, and mask.

[0044] S102: Perform multi-stage downsampling on the source point surface point cloud and the reference surface point cloud respectively, and perform feature extraction on the obtained multi-stage point clouds to generate source point cloud superpoints and corresponding source point cloud dense points, reference point cloud superpoints and corresponding reference point cloud dense points, source point cloud multi-scale features, and reference point cloud multi-scale features; the dense points include the point cloud obtained by the first-stage downsampling, and the superpoints include the point cloud obtained by the last-stage downsampling; in specific implementation, the embodiment of the present application can provide using a multi-stage network based on kernel point convolution for feature extraction to obtain the source point cloud multi-scale features and the reference point cloud multi-scale features.

[0045] S103: Input the source point cloud superpoints, the reference point cloud superpoints, the source point cloud superpoint features, and the reference point cloud superpoint features into a superpoint matching module based on the geometric Transformer architecture to obtain the corresponding relationship between the source point cloud superpoints and the reference point cloud superpoints; in specific implementation, the embodiment of the present application can provide that the superpoint matching module includes geometric structure embedding, self-attention, cross-attention, and feature matching; the geometric structure embedding is used to abstract the superpoints into special encodings, the self-attention is used to learn the global connections of the superpoints of a single point cloud in the geometric space and the feature space, the cross-attention is used to further learn the global connections between the superpoints of the source point cloud and the reference point cloud, so as to generate mixed features to represent these connections, and the feature matching also calculates the feature distance and the Gaussian correlation matrix to obtain a series of alternative corresponding relationships, and uses the topk strategy to select the optimal corresponding relationship among them.

[0046] S104: The correspondence between the source point cloud superpoints and the reference point cloud superpoints is extended by the optimal transport method to obtain the correspondence between the source point cloud dense points and the reference point cloud dense points. Specifically, in implementation, the embodiments of the present application may provide the optimal transport method, input the dense point features and the superpoint correspondence, and the result is to output a cost matrix. The cost matrix is processed by the sinkhorn algorithm and transformed into a soft assignment matrix. The soft assignment matrix is processed according to the confidence level to generate a confidence matrix. The confidence matrix and the multiple top-k strategy are used to obtain the correspondence between the dense points.

[0047] S105: The weighted singular value decomposition is used for the correspondence between the dense points in each region to obtain the transformation matrix within the region. All the obtained different transformation matrices are globally optimized to obtain a unique target transformation matrix, so as to apply the target transformation matrix to the source point cloud and the reference point cloud to complete the registration.

[0048] Specifically, in implementation, the embodiments of the present application may provide that the weighted singular value decomposition includes multiple transformation matrices calculated according to the dense points corresponding to each pair of superpoints. A global optimization function is used to calculate the transformation matrix that maximizes the inlier rate of all the dense points of the registered source point cloud and reference point cloud as the target transformation matrix used for the registration of the source point cloud and the reference point cloud.

[0049] The knee joint point cloud registration method provided by the embodiments of the present application will be introduced in detail below.

[0050] Step 1: The knee joint bone surface point cloud obtained by CT scan reconstruction before surgery (referred to as the source point cloud) and the knee joint bone surface point cloud obtained by various point cloud acquisition devices during surgery (referred to as the reference point cloud) are respectively subjected to shape topology information extraction and superposition to generate a surface point cloud containing shape topology information.

[0051] Specifically, in implementation, the shape topology information of the source point cloud and the reference point cloud is extracted, and the information is superposed back to the original point cloud.

[0052] In this embodiment, the source point cloud and the reference point cloud need to be specially processed first to extract the shape topology information of the point cloud and superpose it back to the original point cloud in forms including but not limited to color, label, and mask.

[0053] This step uses Figure 2 the network module mentioned for processing, referring to Figure 2, this module contains structures such as a feature encoder, a flow predictor, a flow generator, and a point generator. The meaning of the flow in the flow generator can be regarded as shape topology information. This module encodes the point cloud through the feature encoder to obtain a feature vector, inputs the feature vector into the flow generator, inputs the point cloud and the feature vector into the flow predictor. The flow generator and the flow predictor will cooperate to generate a label, which contains information about which part each point should be assigned to. Inputting the label, the feature vector, and the point cloud into the point generator can output the surface point cloud containing shape topology information.

[0054] This module is a relatively independent deep learning network and needs to be pre-trained before the registration network training to reduce the difficulty of the registration network training. The number of flows is an important hyperparameter of this module, and different types of point clouds require different numbers of flows.

[0055] Step 2: Perform multi-stage downsampling and feature extraction on the processed source point cloud and reference point cloud to generate superpoints and corresponding dense points and their features. Specifically, when implementing, perform multi-stage downsampling on the above-mentioned surface point cloud containing shape topology information, and change to a deeper sampling depth after each stage of downsampling is completed. Use the point cloud obtained in the first stage as the dense points, and the point cloud in the last stage as the superpoints.

[0056] Process the dense points and superpoints together to construct the correspondence between the superpoints and the dense points. Each superpoint corresponds to a certain number of dense points. Use a mask to record which superpoint the dense point is assigned to and whether the assignment is valid. At this time, the dense points assigned to the same superpoint are called nodes.

[0057] Extract features from the multi-stage point cloud obtained above to obtain the multi-scale features of the point cloud.

[0058] Perform multi-stage voxel downsampling and feature extraction on the previously obtained point cloud containing shape topology information. Here, after each stage, the voxel size of the voxel downsampling will double for deeper downsampling. The point cloud sampled in the first stage is used as the dense points, and the points sampled in the last stage are used as the superpoints. Place the points sampled in each stage in a list and use a multi-stage network based on kernel point convolution for feature extraction to obtain the features of the superpoints and the dense points. Here, the number of stages matches the number of stages of the voxel downsampling.

[0059] Step 3: Process the superpoints with geometric structure embedding, input the result and the superpoint features into a geometric Transformer to learn the internal relationship, obtain the fused features, and then perform matching to obtain the correspondence of the superpoints. Specifically, when implementing, input the superpoints and their feature vectors of the source point cloud and the reference point cloud into the network module of the geometric Transformer architecture, and use self-attention and cross-attention inside the network to learn the relationships between these superpoints and features, and obtain the fused feature vectors containing these relationships.

[0060] Match the fusion feature vectors of the superpoints of the source point cloud and the reference point cloud to obtain the corresponding relationships between the superpoints.

[0061] Use Geometric Structure Embedding to process the superpoints, and input the encoded results and the features of the superpoints into the geometric Transformer for learning. Learn the internal relationships to obtain the fusion features, and then perform matching to obtain the corresponding relationships of the superpoints.

[0062] This step uses Figure 3 the mentioned superpoint matching module based on the geometric Transformer, referring to Figure 3 , this module has parts such as geometric structure embedding, self-attention, cross-attention, and feature matching. Among them, self-attention and cross-attention together are the geometric Transformer. Geometric structure embedding is used to abstract the superpoints into special encodings, and its core lies in encoding the geometric features of the superpoints with rotational invariance. There is a hyperparameter σd here, which is used to control the sensitivity of the geometric structure embedding to distance changes. For point clouds of different scales, this parameter is used to adjust the encoding effect. Self-attention can learn the global relationships of the superpoints of a single point cloud in the geometric space and the feature space, and cross-attention further learns the global relationships between the superpoints of the source point cloud and the reference point cloud, thereby generating a mixed feature to represent these relationships. This mixed feature is invariant to transformations and robust to inferring corresponding relationships. Both the source point cloud and the reference point cloud have mixed features for their superpoints. Matching their mixed features can obtain the corresponding relationships. Here, the matching method used is Gaussian correlation. By calculating the feature distance and the Gaussian correlation matrix, a series of possible corresponding relationships are obtained, and the topk strategy is used to select the optimal corresponding relationships among them.

[0063] Step 4: Extend the corresponding relationships between the superpoints to the corresponding relationships between the dense points through methods such as optimal transport. When specifically implemented, for each pair of corresponding superpoints, use optimal transport to establish multiple corresponding relationships between the dense points assigned to these superpoints.

[0064] In each pair of nodes, use the confidence matrix and the multiple topk strategy to select the most reasonable corresponding relationship among these corresponding relationships.

[0065] After obtaining the correspondence of superpoints, methods such as optimal transport are used to extend it to the correspondence between dense points. The correspondence between dense points constructed here refers to the correspondence between the dense points corresponding to each pair of superpoints. Optimal transport will find a transport plan to minimize the cost of transforming one distribution into another. The output result of this method is a cost matrix, which is processed by the sinkhorn algorithm and transformed into a soft assignment matrix. A confidence matrix is generated based on the confidence level for the soft assignment matrix, and the correspondence between dense points is obtained using the confidence matrix and the multiple topk strategy.

[0066] Step 5: Use weighted singular value decomposition for the correspondence between dense points in each region to obtain the transformation matrix within the region and optimize it into the transformation matrix for the overall point cloud. In specific implementation, for the nodes with established correspondence relationships, the weighted singular value decomposition method is used to obtain the transformation matrix.

[0067] For the different transformation matrices obtained for all node pairs, a unique transformation matrix is obtained through global optimization, and then this transformation matrix is applied to the entire source point cloud and reference point cloud to complete the registration.

[0068] For the dense points within the corresponding region of each pair of superpoints, according to their correspondence relationships, the weighted singular value decomposition method is used to calculate the transformation matrix. The correspondence relationships between several dense points obtained through the topk strategy have their respective correspondence confidence scores, and this score will be used as the weight for the weighted singular value decomposition to calculate the optimal transformation matrix. For the multiple transformation matrices calculated based on the dense points corresponding to each pair of superpoints, a global optimization function is used to calculate the transformation matrix that can maximize the inlier rate of all dense points of the registered source point cloud and reference point cloud as the final transformation matrix of the source point cloud and reference point cloud.

[0069] In summary, the knee joint point cloud registration method provided in this application can register different modality knee joint point clouds reconstructed from CT (Computed Tomography) images and collected by point cloud acquisition devices, and is used for guiding knee joint replacement surgery. It can effectively achieve the registration between the preoperative CT scan reconstructed point cloud and the point cloud collected by the point cloud acquisition device, effectively process the special shape information of the above-mentioned point clouds, overcome the difficulty of low overlap rate existing in the above-mentioned point clouds, and provide accurate information for augmented reality surgical navigation.

[0070] See Figure 4 , this embodiment of the application can also provide a knee joint point cloud registration device, as Figure 4 shown, for executing the above-mentioned knee joint point cloud registration method. The device may include:

[0071] The point cloud topological shape information generation unit 401 is configured to obtain a source point cloud and a reference point cloud. The source point cloud includes the knee joint bone surface point cloud obtained by preoperative CT scan reconstruction, and the reference point cloud includes the knee joint bone surface point cloud obtained by a point cloud acquisition device during the operation. The point cloud shape topological information generation module is used to extract and superimpose the shape topological information of the source point cloud and the reference point cloud respectively, and generate a source point surface point cloud and a reference surface point cloud containing shape topological information.

[0072] The feature extraction unit 402 is configured to perform multi-stage downsampling on the source point surface point cloud and the reference surface point cloud respectively, and extract features from the obtained multi-stage point clouds to generate source point cloud superpoints and corresponding source point cloud dense points, reference point cloud superpoints and corresponding reference point cloud dense points, source point cloud multi-scale features, and reference point cloud multi-scale features. The dense points include the point cloud obtained by the first-stage downsampling, and the superpoints include the point cloud obtained by the last-stage downsampling.

[0073] The superpoint matching unit 403 is configured to input the source point cloud superpoints, the reference point cloud superpoints, the source point cloud superpoint features, and the reference point cloud superpoint features into the superpoint matching module based on the geometric Transformer architecture to obtain the corresponding relationship between the source point cloud superpoints and the reference point cloud superpoints.

[0074] The global point matching unit 404 is configured to expand the corresponding relationship between the source point cloud superpoints and the reference point cloud superpoints through the optimal transport method to obtain the corresponding relationship between the source point cloud dense points and the reference point cloud dense points.

[0075] The local-to-global registration unit 405 is configured to use weighted singular value decomposition for the corresponding relationship between the dense points in each region to obtain the transformation matrix within the region. All the obtained different transformation matrices are globally optimized to obtain a unique target transformation matrix, so as to apply the target transformation matrix to the source point cloud and the reference point cloud to complete the registration.

[0076] An embodiment of the present application can also provide a knee joint point cloud registration device, which includes a processor and a memory:

[0077] The memory is used to store program codes and transmit the program codes to the processor;

[0078] The processor is configured to execute the steps of the above-mentioned knee joint point cloud registration method according to the instructions in the program codes.

[0079] Such as Figure 5As shown in the figure, a knee joint point cloud registration device provided by an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 all complete mutual communication through the communication bus 13.

[0080] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic devices, etc.

[0081] The processor 10 may call the program stored in the memory 11. Specifically, the processor 10 may execute the operations in the embodiment of the knee joint point cloud registration method.

[0082] The memory 11 is used to store one or more programs. The program may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 stores at least programs for implementing the following functions:

[0083] Obtain a source point cloud and a reference point cloud. The source point cloud includes the knee joint bone surface point cloud obtained by preoperative CT scan reconstruction, and the reference point cloud includes the knee joint bone surface point cloud obtained by a point cloud acquisition device during the operation; use the point cloud shape topology information generation module to respectively extract and superimpose the shape topology information of the source point cloud and the reference point cloud to generate a source point surface point cloud and a reference surface point cloud containing shape topology information;

[0084] Perform multi-stage downsampling on the source point surface point cloud and the reference surface point cloud respectively, and perform feature extraction on the obtained multi-stage point clouds to generate source point cloud superpoints and corresponding source point cloud dense points, reference point cloud superpoints and corresponding reference point cloud dense points, source point cloud multi-scale features, and reference point cloud multi-scale features; the dense points include the point cloud obtained by the first-stage downsampling, and the superpoints include the point cloud obtained by the last-stage downsampling;

[0085] Input the source point cloud superpoints, the reference point cloud superpoints, the source point cloud superpoint features, and the reference point cloud superpoint features into the superpoint matching module based on the geometric Transformer architecture to obtain the corresponding relationship between the source point cloud superpoints and the reference point cloud superpoints;

[0086] Expand the corresponding relationship between the source point cloud superpoints and the reference point cloud superpoints through the optimal transport method to obtain the corresponding relationship between the source point cloud dense points and the reference point cloud dense points;

[0087] The corresponding relationship between each pair of regional dense points is used to obtain an intra-region transformation matrix through weighted singular value decomposition; all the different obtained transformation matrices are globally optimized to obtain a unique target transformation matrix, so as to apply the target transformation matrix to the source point cloud and the reference point cloud to complete registration.

[0088] In a possible implementation manner, the memory 11 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area may store data created during use, such as initialization data, etc.

[0089] In addition, the memory 11 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage devices.

[0090] The communication interface 12 may be an interface of a communication module for connecting to other devices or systems.

[0091] Of course, it should be noted that Figure 5 the structure shown does not constitute a limitation on the knee joint point cloud registration device in the embodiments of the present application. In practical applications, the knee joint point cloud registration device may include more or fewer components than Figure 5 those shown, or combine certain components.

[0092] Embodiments of the present application may also provide a computer-readable storage medium for storing program code, and the program code is used to execute the steps of the above-mentioned knee joint point cloud registration method.

[0093] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0094] As can be seen from the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0095] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. The key point of each embodiment is to illustrate the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiment. The systems and system embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0096] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A knee joint point cloud registration method, characterized in that, Including: Obtain a source point cloud and a reference point cloud, where the source point cloud includes the surface point cloud of the knee joint bone obtained by preoperative CT scan reconstruction, and the reference point cloud includes the surface point cloud of the knee joint bone obtained by a point cloud acquisition device during the operation; use the point cloud shape topology information generation module to respectively extract and superimpose the shape topology information of the source point cloud and the reference point cloud to generate a source point surface point cloud and a reference surface point cloud containing shape topology information; Perform multi-stage downsampling on the source point surface point cloud and the reference surface point cloud respectively, and extract features from the obtained multi-stage point clouds to generate source point cloud superpoints and corresponding source point cloud dense points, reference point cloud superpoints and corresponding reference point cloud dense points, source point cloud multi-scale features and reference point cloud multi-scale features; the dense points include the point cloud obtained by the first-stage downsampling, and the superpoints include the point cloud obtained by the last-stage downsampling; Input the source point cloud superpoints, the reference point cloud superpoints, the source point cloud superpoint features and the reference point cloud superpoint features into the superpoint matching module based on the geometric Transformer architecture to obtain the corresponding relationship between the source point cloud superpoints and the reference point cloud superpoints; Expand the corresponding relationship between the source point cloud superpoints and the reference point cloud superpoints through the optimal transport method to obtain the corresponding relationship between the source point cloud dense points and the reference point cloud dense points; Use weighted singular value decomposition for the corresponding relationship between dense points in each region to obtain the transformation matrix within the region; Through global optimization, obtain a unique target transformation matrix from all the obtained different transformation matrices, so as to apply the target transformation matrix to the source point cloud and the reference point cloud to complete registration.

2. The knee joint point cloud registration method according to claim 1, wherein The point cloud shape topology information generation module includes a feature encoder, a flow predictor, a flow generator and a point generator; the feature encoder is used to encode the point cloud to obtain a feature vector, and the flow generator and the flow predictor are used to jointly generate a label using the point cloud and the feature vector, and the label includes information on which part each point should be assigned to; the point generator is used to generate a surface point cloud containing shape topology information according to the label, the feature vector and the point cloud.

3. The knee joint point cloud registration method according to claim 1, wherein The superimposition includes superimposing the shape topology information back into the original point cloud in any one form or a combination of several forms such as color, label, and mask.

4. The knee joint point cloud registration method according to claim 1, wherein Use a multi-stage network based on kernel point convolution for feature extraction to obtain the source point cloud multi-scale features and the reference point cloud multi-scale features.

5. The knee joint point cloud registration method according to claim 1, characterized in that The superpoint matching module includes geometric structure embedding, self-attention, cross-attention and feature matching; the geometric structure embedding is used to abstract the superpoints into special encodings, the self-attention is used to learn the global connection of the superpoints of a single point cloud in the geometric space and the feature space, the cross-attention is used to further learn the global connection between the superpoints of the source point cloud and the reference point cloud, so as to generate mixed features to characterize these connections, and the feature matching also obtains a series of alternative corresponding relationships by calculating the feature distance and the Gaussian correlation matrix, and selects the optimal corresponding relationship among them using the topk strategy.

6. The knee joint point cloud registration method according to claim 1, characterized in that The result of the optimal transport method is to output a cost matrix, which is processed by the sinkhorn algorithm and transformed into a soft assignment matrix. A confidence matrix is generated by processing the soft assignment matrix according to the confidence. The correspondence between dense points is obtained by using the confidence matrix and the multiple top-k strategies.

7. The knee joint point cloud registration method according to claim 1, wherein The weighted singular value decomposition includes multiple transformation matrices calculated based on the dense points corresponding to each pair of superpoints. A global optimization function is used to calculate the transformation matrix that maximizes the inlier rate of all dense points of the registered source point cloud and reference point cloud, which is used as the target transformation matrix for registering the source point cloud and the reference point cloud.

8. A knee joint point cloud registration device, characterized in that, A device for performing the knee joint point cloud registration method according to any one of claims 1-7, the device comprising: A point cloud topological shape information generation unit for obtaining a source point cloud and a reference point cloud. The source point cloud includes the knee joint bone surface point cloud obtained by preoperative CT scan reconstruction, and the reference point cloud includes the knee joint bone surface point cloud obtained by a point cloud acquisition device during surgery. The point cloud shape topological information generation module is used to extract and superimpose the shape topological information of the source point cloud and the reference point cloud respectively, and generate a source surface point cloud and a reference surface point cloud containing shape topological information. A feature extraction unit for performing multi-stage downsampling on the source surface point cloud and the reference surface point cloud respectively, and extracting features from the obtained multi-stage point clouds to generate source point cloud superpoints and corresponding source point cloud dense points, reference point cloud superpoints and corresponding reference point cloud dense points, source point cloud multi-scale features, and reference point cloud multi-scale features. The dense points include the point cloud obtained by the first-stage downsampling, and the superpoints include the point cloud obtained by the last-stage downsampling. A superpoint matching unit for inputting the source point cloud superpoints, the reference point cloud superpoints, the source point cloud superpoint features, and the reference point cloud superpoint features into a superpoint matching module based on the geometric Transformer architecture to obtain the correspondence between the source point cloud superpoints and the reference point cloud superpoints. A global point matching unit for expanding the correspondence between the source point cloud superpoints and the reference point cloud superpoints through the optimal transport method to obtain the correspondence between the source point cloud dense points and the reference point cloud dense points. A local-to-global registration unit for obtaining the intra-region transformation matrix for the correspondence between dense points in each region by using weighted singular value decomposition. All the obtained different transformation matrices are globally optimized to obtain a unique target transformation matrix, so as to apply the target transformation matrix to the source point cloud and the reference point cloud to complete the registration.

9. A knee joint point cloud registration device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the knee joint point cloud registration method according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code, and the program code is used to execute the knee joint point cloud registration method according to any one of claims 1-7.

Images