Deep Learning-Based Registration Method and System for 3D Multi-Module Medical Images
By using deep learning methods to preprocess, extract feature points, and optimize registration of 3D medical images, the problem of registration of images with different modalities is solved, and accurate registration and diagnostic assistance of multimodal images are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2023-02-14
- Publication Date
- 2026-05-26
AI Technical Summary
Existing technologies struggle to effectively combine and register 3D medical images of different modalities, resulting in a lack of effective reference data for medical staff during diagnosis.
A deep learning-based approach is adopted to preprocess point cloud data, extract feature points, perform pre-registration and optimization, and then use threshold point cloud registration and point cloud segmentation registration for accurate registration.
It achieves precise registration of medical images of different modalities in the same region, provides better diagnostic reference, supports multi-angle observation and operation, and improves the accuracy and efficiency of diagnosis.
Smart Images
Figure CN116128942B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning in three-dimensional medical imaging, and more specifically, to a registration method and system for three-dimensional multi-module medical images based on deep learning. Background Technology
[0002] Deep learning is a subfield of machine learning and a frequently used method in the field of artificial intelligence. Originating from research on artificial neural networks, deep learning learns the inherent patterns and representational layers of large-scale data by combining low-level features to construct more abstract high-level features. The ultimate goal is to enable machines to possess human-like analytical and learning capabilities, capable of recognizing and processing data such as text, images, and sound. Deep learning achieves its learning objectives and effects by establishing an appropriate number of neural computation nodes and a hierarchical structure with multiple operations, selecting suitable inputs and outputs, and optimizing the deep learning network.
[0003] Currently, with the continuous improvement of medical standards and the advent of the big data era, in addition to traditional text and image information for diagnosis, the analysis and application of medical imaging data has become one of the core aspects of clinical medical diagnosis. Medical staff need to manually interpret this data to assist in the diagnosis of patients. Furthermore, with the development of internet technology, more and more technologies are transitioning to the three-dimensional field. Multimodal medical imaging can simultaneously display different data in the same three-dimensional space, providing more effective reference data for further diagnosis by medical staff. Summary of the Invention
[0004] In view of the deficiencies in the existing technology, the purpose of this invention is to provide a registration method and system for three-dimensional multi-module medical images based on deep learning.
[0005] According to one aspect of the present invention, a registration method for three-dimensional multi-module medical images based on deep learning is provided, comprising:
[0006] Acquire point cloud data of medical images of different modalities and perform preprocessing;
[0007] Feature points of different modalities of medical images are extracted from preprocessed point cloud data using deep learning networks;
[0008] The feature points are pre-registered to determine the output positions of the different modal medical images;
[0009] The pre-registration results are optimized by threshold point cloud registration and point cloud segmentation registration.
[0010] Preferably, the preprocessing includes:
[0011] Using pre-set prior information, three-dimensional mesh data of different human tissues are extracted and reconstructed from the point cloud data;
[0012] The three-dimensional mesh data is smoothed, centered, and discrete points are removed.
[0013] The point cloud data is downsampled.
[0014] Preferably, the preprocessing further includes:
[0015] The point cloud data and corresponding feature point labels are synchronously and randomly rotated in three dimensions to generate multiple augmented data in batches.
[0016] Preferably, the deep learning network performs machine learning on the downsampled point cloud data to extract the three-dimensional spatial coordinates of specified feature points of the three-dimensional medical images for each modality.
[0017] Preferably, the specified feature points are feature points in the common area of medical images of different modalities, selected through medical analysis, and have the characteristic that their relative positions remain unchanged under different expressions and postures.
[0018] Preferably, the pre-registration includes:
[0019] Identify the same feature points in different modalities;
[0020] Based on the same feature points, a matrix factorization method is used to learn a preliminary non-rigid affine transformation;
[0021] The non-rigid affine transformation includes a rotation matrix and a translation matrix, and the process of obtaining each of them includes:
[0022] For two initial feature point sets A and B, define B = R*A + t, where R is the rotation matrix and t is the translation matrix:
[0023] Decentralize the two sets of points by subtracting the mean of the corresponding set from each point to obtain new sets of points A' and B'.
[0024] Calculate the covariance matrix H:
[0025]
[0026] N is the total number of point clouds in point cloud datasets A or B. Represents the i-th point in point set A, and similarly... μ represents the i-th point in the set of points B; A and μ BA and B are the mean values of point sets A and B, respectively; A' and B' are the results of decentralizing the two original point cloud data, that is, subtracting the mean from the original data. This step can better analyze the relative coordinate relationship between point clouds rather than just the absolute size of the point cloud data.
[0027] Find the orthogonal matrix U, the nonnegative real diagonal matrix S, and the orthogonal matrix V using singular value decomposition (SVD):
[0028] [U,S,V]=SVD(H)
[0029] Calculate the rotation and translation matrices:
[0030] R = VU T
[0031] t=-R×μ A +μ B .
[0032] Preferably, the registration optimization includes point cloud segmentation registration and region segmentation registration;
[0033] The point cloud segmentation and registration method modifies the distance threshold in the point cloud registration method to filter feature points smaller than the distance threshold.
[0034] The region segmentation and registration process involves segmenting the feature points extracted by the deep learning network into feature regions, obtaining the point cloud distribution of the feature regions, and adjusting the registration weights of the feature regions.
[0035] According to a second aspect of the present invention, a registration system for three-dimensional multi-module medical images based on deep learning is provided, comprising:
[0036] A data sampling generation block acquires point cloud data of medical images of different modalities and performs preprocessing.
[0037] The feature point learning block extracts feature points of different modalities of medical images from preprocessed point cloud data through a deep learning network.
[0038] A pre-registration block, which pre-registers the feature points to determine the output positions of the different modal medical images;
[0039] The calibration block optimizes the pre-registration result through threshold point cloud registration and point cloud segmentation registration.
[0040] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform the methods described above, or to run the systems described above.
[0041] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the methods described above, or to run the system described above.
[0042] Compared with the prior art, the present invention has the following beneficial effects:
[0043] The deep learning-based three-dimensional multi-module medical image registration method and system in this invention combines medical images of different modalities and performs three-dimensional registration and optimization, which can display medical images of different modalities in the same area to assist medical staff in making better references.
[0044] The deep learning-based registration method and system for three-dimensional multi-modal medical images in this invention can be applied to the registration of different combinations of medical images. Feature point learning can be dynamically adjusted according to different scenarios and needs, and anatomically significant feature points can be set in different medical fields. Correspondingly, the point cloud segmentation part can also be dynamically adjusted according to the needs of different scenarios to achieve better diagnostic reference results.
[0045] The deep learning-based three-dimensional multi-module medical image registration method and system in this invention provides multi-modal registration results that can assist medical staff in making better references on interactive software. The multi-modal medical image registration results, stored in a three-dimensional mesh format, can be manually rotated and scaled to support operation and observation at different angles and depths. The fused data of the multi-modal medical images can better reflect the overall information of the patient's facial contours, structure, and texture. Based on this, medical staff can observe rich details of the target area through simple operations such as scaling, cutting, and removing occlusions in the system displaying the registration results. This allows for both macroscopic observation of the overall information of the target area and detailed diagnostic reference. Attached Figure Description
[0046] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0047] Figure 1 This is a flowchart of a deep learning-based registration method for three-dimensional multi-module medical images according to an embodiment of the present invention.
[0048] Figure 2 This is a framework diagram of a deep learning-based registration method for three-dimensional multi-module medical images according to another embodiment of the present invention.
[0049] Figure 3This is a structural diagram of a deep learning-based three-dimensional multi-module medical image registration system according to an embodiment of the present invention. Detailed Implementation
[0050] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.
[0051] like Figure 1 The diagram shown is a flowchart illustrating a deep learning-based three-dimensional multimodal medical image registration method according to the present invention, comprising:
[0052] S100 acquires point cloud data of medical images of different modalities and performs preprocessing;
[0053] S200 uses a deep learning network to extract feature points of different modalities of medical images from the point cloud data preprocessed by S100.
[0054] S300 pre-registers the feature points extracted by S200 to determine the output position of different modal medical images;
[0055] S400 optimizes the pre-registration results of S400 by using threshold point cloud registration and point cloud segmentation registration.
[0056] Different modalities of medical images have different imaging effects and imaging areas, and their initial three-dimensional spatial positions are also far apart. This embodiment combines medical images from different modalities and performs three-dimensional registration, then displays them in the same area to help medical staff make better references.
[0057] Because there are too many original data points, the computational requirements for feature point learning are too high, so the original data needs to be processed. Therefore, in a preferred embodiment of the present invention, S100 is implemented to preprocess the collected point cloud data. Specifically, three-dimensional mesh data of different human tissues (such as skin and bones) are extracted and reconstructed through pre-set prior information, and the three-dimensional mesh data is downsampled, including smoothing, centering, and removing discrete points.
[0058] In practice, different types of data have different prior information. Taking a common CT scan as an example, prior information includes, but is not limited to, the HU value boundary of the CT scan, which is used to segment different tissues of the human body through thresholding. By downsampling, the number of data points is reduced, thereby reducing the computational requirements for feature point learning.
[0059] Since the direct reconstruction results may contain discrete points and special protrusions, the reconstructed 3D mesh data is "refined" by using operations such as smoothing, centering, and discrete point removal to make the reconstructed 3D mesh results more consistent with reality.
[0060] In another embodiment of the present invention, S200 is implemented, in which the downsampled data of the original data and the feature point information are put into the deep neural network for learning after obtaining the downsampled data in S100. The resulting deep neural network can automatically generate the corresponding feature key points after obtaining a new downsampled data.
[0061] Furthermore, when raw medical image data is scarce, the aforementioned deep neural learning network can use random three-dimensional spatial rigid rotation transformations to enhance the raw data and obtain more data.
[0062] In this embodiment, the number of initial downsampled data points fed into the deep neural network for feature point learning can be dynamically adjusted according to the actual computing power and the requirements of the model. This means the deep neural network has generalization and portability; for registration of different tissues (such as the face, lungs, etc.), the feature points can be dynamically adjusted and trained using corresponding data. In other words, feature point learning can be dynamically adjusted according to different scenarios and needs, and anatomically significant feature points can be set in different medical fields.
[0063] In a preferred embodiment of the present invention, after feature points are extracted using a deep learning network, the corresponding feature points under different modalities are pre-registered. In this embodiment, the pre-registration learns a preliminary non-rigid affine transformation based on the feature points by calling a matrix factorization method.
[0064] Specifically, a preliminary non-rigid affine transformation is learned by calling a matrix factorization method based on the same feature points;
[0065] The non-rigid affine transformation includes a rotation matrix and a translation matrix, and the process of obtaining each of them includes:
[0066] For two initial feature point sets A and B, define B = R*A + t, where R is the rotation matrix and t is the translation matrix:
[0067] Decentralize the two sets of points by subtracting the mean of the corresponding set from each point to obtain new sets of points A' and B'.
[0068] Calculate the covariance matrix H:
[0069]
[0070] N is the total number of point clouds in point cloud datasets A or B. Represents the i-th point in point set A, and similarly... μ represents the i-th point in the set of points B; A and μ B A and B are the mean values of point sets A and B, respectively; A' and B' are the results of decentralizing the two original point cloud data, that is, subtracting the mean from the original data. This step can better analyze the relative coordinate relationship between point clouds rather than just the absolute size of the point cloud data.
[0071] Find the orthogonal matrix U, the nonnegative real diagonal matrix S, and the orthogonal matrix V using singular value decomposition (SVD):
[0072] [U,S,V]=SVD(H)
[0073] Calculate the rotation and translation matrices:
[0074] R = VU T
[0075] t=-R×μ A +μ B .
[0076] Singular Value Decomposition (SVD) is a linear algebraic method that decomposes a large original matrix into several submatrices to obtain the transformation relationship between the matrices. Here, SVD is used to obtain the rotation matrix between two point cloud datasets. U, S, and V are the three submatrices obtained after SVD decomposition, where U and V are orthogonal matrices, and S is a non-negative real diagonal matrix. The final rotation matrix is R = VU. T .
[0077] Even after pre-registration, slight discrepancies still exist between different modal medical images, requiring further correction. In a preferred embodiment of the invention, step S400 is implemented for further correction. Threshold point cloud registration and point cloud segmentation registration are employed. The correction process is as follows: the pre-registration result is first segmented into common information regions using a point cloud segmentation method (because the information contained in different modal data is not entirely the same, for example, facial scans may contain hair information while CT scans may not); based on this, threshold point cloud registration is performed, limiting threshold registration to allow the registration method to focus on effective registration regions with high similarity. The purpose of both is to reduce the impact of invalid regions on the registration accuracy of the image.
[0078] Threshold point cloud registration filters initial data points within a certain distance by modifying the distance threshold in the point cloud registration method, thereby reducing the impact of points with large deviations. The specific process of point cloud segmentation and registration involves using the ICP (Iterative ClosetPoint) method to perform fine-grained registration on the pre-segmented point cloud. Since the convergence threshold set in the ICP method affects the iteration process, a larger threshold is initially set to ensure the method can search for more informative surrounding points. Subsequently, the threshold is decreased to ensure the method focuses more on nearby points. The ICP method is a data registration method based on nearest-point search; the threshold in the method indicates that only feature points within a threshold radius are considered for a given point.
[0079] The point cloud segmentation method involves cutting 3D data according to different scenes and requirements, thereby increasing the registration weight of important regions. The specific process of feature region segmentation involves obtaining candidate feature regions through point cloud segmentation, and then calculating the registration of different feature regions using the aforementioned point cloud segmentation method. Finally, error analysis is performed on the registration results, increasing the weight ratio of feature regions with smaller errors in the registration process, while correspondingly decreasing the weight ratio of feature regions with larger errors.
[0080] The point cloud segmentation method in this embodiment includes, but is not limited to, ICP.
[0081] In the above embodiments, based on different modalities of 3D medical images and different combined registration methods, the number and position of feature points can be dynamically adjusted according to anatomical significance to achieve better registration results. After registration is completed, the registration effect under different modalities can be directly observed, and the embodiments of the present invention can also be displayed in a subsequent integrated display system. The registration results of multimodal medical images stored in the form of a 3D mesh can be manually rotated and scaled to support operation and observation at different angles and depths. At the same time, the fused data of multimodal medical images can better reflect the overall information of the patient's facial contours, structures, and textures. On this basis, medical staff can observe the rich details of the target area through simple operations such as scaling, cutting, and removing occlusions in the system displaying the registration results. This can assist medical staff in observing the overall information of the target area macroscopically, and can also provide diagnostic reference for the target area in detail.
[0082] like Figure 2As shown, another embodiment of the present invention is presented. The registration method first divides medical images of different modalities into a training set and a test set. The training set data is standardized with artificial feature points (the number and location of feature points are determined manually), and the feature point information and the downsampled point cloud data are fed together into a deep neural network for learning and optimization. The test set data can be used as new data to verify the deep learning network's ability to learn feature points. Feature points from different modalities in the new data are combined and calculated, and matrix factorization is used to obtain a preliminary affine transformation matrix to adjust the positions of the 3D data of different modalities. Subsequent preset point cloud registration blocks and point cloud segmentation registration blocks further refine the preliminary registration to achieve better registration results.
[0083] Based on the same inventive concept, in other embodiments of the present invention, a registration system for three-dimensional multi-module medical images based on deep learning is provided, see [link to relevant documentation]. Figure 3 The system includes a data sampling and generation block, a feature point learning block, a pre-registration block, and a calibration block. The data sampling and generation block acquires point cloud data of different modalities of medical images and performs preprocessing. The feature point learning block extracts feature points of different modalities of medical images from the preprocessed point cloud data using a deep learning network. The pre-registration block performs pre-registration on the feature points to determine the output position of the different modalities of medical images. The calibration block optimizes the pre-registration results through threshold point cloud registration and point cloud segmentation registration.
[0084] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the above-described method or to run the above-described system.
[0085] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the above-described method or to run the above-described system.
[0086] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.
Claims
1. A method for registering three-dimensional multi-modal medical images based on deep learning, characterized in that, include: S1, acquire point cloud data of medical images of different modalities and perform preprocessing; The preprocessing includes: Based on prior knowledge of human anatomy, a three-dimensional mesh model corresponding to the tissue type is extracted and reconstructed from the point cloud data. The three-dimensional mesh model is smoothed, spatially centered, and outlier points are removed. The point cloud data is downsampled; The point cloud data and its corresponding feature point labels are then subjected to synchronous random three-dimensional rotation to generate amplified training samples in batches. S2, extract common region feature points from preprocessed point cloud data using a deep learning network, wherein the feature points are anatomical landmarks whose relative positions remain stable under different expressions or postures; S3, pre-register the feature points to determine the initial output positions of the different modal medical images; S4, the pre-registration result is finely optimized by threshold point cloud registration and point cloud segmentation registration to obtain the final registration result; Specifically, the threshold point cloud registration involves modifying the distance threshold in the point cloud registration method from large to small to filter feature points smaller than the distance threshold; the threshold indicates that for a given point, only feature points within the threshold radius are considered. The point cloud segmentation and registration process involves segmenting the feature points extracted by the deep learning network into feature regions, obtaining the point cloud distribution of the feature regions, and adjusting the registration weights of the feature regions.
2. The method of claim 1, wherein, The feature points mentioned in S2 are feature points of the common area of medical images of different modalities. They are selected through medical analysis and have the characteristic that their relative positions remain unchanged under different expressions and postures. 3.The method of claim 1, wherein, The pre-registration includes: Identify the same feature points in different modalities; Based on the same feature points, a matrix factorization method is used to learn a preliminary non-rigid affine transformation; The non-rigid affine transformation includes a rotation matrix and a translation matrix, and the process of obtaining each of them includes: For two initial feature point sets A and B, we define B = R * A + t, where R is the rotation matrix and t is the translation matrix: Decentralize the two sets of points by subtracting the mean of the corresponding set from each point to obtain new sets of points A' and B'. Calculate the covariance matrix H: ； N is the total number of point clouds in the point cloud dataset A or B, the i-th point of the A point set, and the same for the i-th point of the B point set; and are the data mean values of the point sets A and B, respectively; A' and B' are the results of the two original point cloud data after decentralization, i.e., the original data minus the mean value. Find the orthogonal matrix U, the nonnegative real diagonal matrix S, and the orthogonal matrix V using singular value decomposition (SVD): ； Calculate the rotation and translation matrices: 。 4. A deep learning based registration system for three-dimensional multi-modal medical images for implementing the method of any one of claims 1-3, characterized in that, include: A data sampling generation block acquires point cloud data of medical images of different modalities and performs preprocessing. The feature point learning block extracts feature points of different modalities of medical images from preprocessed point cloud data through a deep learning network. A pre-registration block, which pre-registers the feature points to determine the output positions of the different modal medical images; The calibration block optimizes the pre-registration result through threshold point cloud registration and point cloud segmentation registration.
5. A terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-3.
6. A computer-readable storage medium having stored thereon a computer program, characterized in that, When executed by a processor, this program can be used to perform the method described in any one of claims 1-3.