Three-dimensional image cross-modality registration method and device, electronic equipment and storage medium

Abstract

The application provides a three-dimensional image cross-modality registration method and device, electronic equipment and a storage medium. The method comprises the following steps: first, high-precision segmentation of a target part is performed, and a multi-modality registration problem is converted into a geometric alignment problem of the same anatomical entity; then, a two-stage point cloud registration strategy from coarse to fine is adopted, a robust initial transformation is obtained through an initial alignment algorithm based on sampling consistency, fine registration is performed through an improved iterative closest point algorithm, and finally, sub-millimeter multi-modality image fusion is realized, thereby laying a solid foundation for precise orthopedic surgery navigation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for cross-modal registration of three-dimensional images. Background Technology

[0002] In precision orthopedic surgery, image-guided surgical navigation systems, through high-precision alignment of real-time anatomical structures with preoperative images, are crucial for enhancing surgical safety and accuracy. Multimodal image fusion is particularly valuable; CT (Computed Tomography) clearly presents bony details, while MRI (Magnetic Resonance Imaging) provides high-quality visualization of soft tissues. Precise registration and fusion of these two technologies provide a composite 3D map of both bone and soft tissue for surgical planning and navigation, achieving information complementarity. Cross-modal 3D image alignment is the core component.

[0003] In recent years, deep learning methods have achieved fast end-to-end registration. Leveraging the powerful feature learning capabilities of convolutional neural networks, rapid, end-to-end registration can be achieved, significantly improving computational efficiency. However, existing methods still have limitations: First, the performance of many learning-based registration networks heavily relies on large-scale, high-quality labeled data, limiting their generalization ability in the medical field where labeled data is scarce. Second, at the implementation level, widely used algorithms such as iterative nearest point and their variants are extremely sensitive to the initial pose of point cloud data and are easily plagued by local optima, leading to inaccurate registration results when registering CT and MRI point clouds with large modal differences and incomplete overlapping areas. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, electronic device, and storage medium for cross-modal registration of three-dimensional images to address the aforementioned technical problems and improve the registration results.

[0005] In a first aspect, embodiments of this application provide a three-dimensional image cross-modal registration method, including: acquiring source point clouds and target point clouds of a target region; performing a first registration of the source point clouds and target point clouds using a sample consistency initial alignment algorithm, and outputting an initial transformation matrix; performing a second registration of the source point clouds and target point clouds using an improved iterative nearest point algorithm and the initial transformation matrix, until a final transformation matrix is ​​output, so as to complete the spatial alignment and fusion of the source point clouds and target point clouds.

[0006] In one embodiment, acquiring source point clouds and target point clouds of a target region includes: acquiring a first modal image and a second modal image of the target region, wherein the first modal image and the second modal image include images acquired by different imaging principles that can provide complementary anatomical structural information; performing target structure segmentation on the first modal image to obtain first segmentation data; performing target structure segmentation on the second modal image to obtain second segmentation data; extracting a first surface triangular mesh from the first segmentation data and uniformly sampling the first surface triangular mesh to generate a source point cloud; and extracting a second surface triangular mesh from the second segmentation data and sampling the second surface triangular mesh to generate a target point cloud.

[0007] In one embodiment, the source point cloud and the target point cloud are first registered using a sample consistency initial alignment algorithm, including: extracting local geometric feature descriptors for each point in the source point cloud and the local geometric feature descriptors for the target point cloud; establishing candidate corresponding point pairs based on the local geometric feature descriptors of the source point cloud and the target point cloud; iteratively optimizing the candidate corresponding point pairs using a random sample consistency framework and a loss function to obtain an initial transformation matrix, which represents the rigid transformation matrix that minimizes the global loss function, and using the initial transformation matrix as the first registration result.

[0008] In one embodiment, establishing candidate corresponding point pairs includes: calculating a first fast point feature histogram for each point in the source point cloud and a second fast point feature histogram for each point in the target point cloud; randomly selecting multiple sampling points from the source point cloud, and ensuring that the Euclidean distance between any two sampling points is greater than a preset threshold; using the first fast point feature histogram of each sampling point as the query basis, performing a nearest neighbor search in the second fast point feature histogram space of the target point cloud, determining candidate corresponding points with similar features for each sampling point, and constructing candidate corresponding point pairs.

[0009] In one embodiment, a second registration of the source point cloud and the target point cloud based on the initial transformation matrix includes: transforming the source point cloud using the initial transformation matrix to obtain an initial aligned point cloud; filtering corresponding point pairs between the initial aligned point cloud and the target point cloud using normal vector angle constraints and local curvature difference constraints to obtain a set of valid corresponding point pairs; solving for the rigid transformation increment on the set of valid corresponding point pairs using a weighted least squares objective function; updating the current source point cloud using the rigid transformation increment until the convergence condition is met to obtain the target transformation matrix, which includes a fine registration transformation matrix composed of all iterative transformation increments, and the fine registration transformation matrix represents the second registration result.

[0010] In one embodiment, in the normal vector angle constraint, the unit normal vector angle of the candidate corresponding point pair is less than a preset angle threshold, and in the local curvature difference constraint, the normalized local curvature absolute value of the candidate corresponding point pair is less than a preset curvature threshold.

[0011] In one embodiment, the loss value of the loss function is calculated as follows: when the error between corresponding point pairs after transformation is less than a preset error threshold, the loss value is calculated by half the square of the error; when the error between corresponding point pairs after transformation is greater than or equal to the preset error threshold, the loss value is obtained by multiplying the preset error threshold by the absolute value of the error and then subtracting half the square of the preset threshold.

[0012] Secondly, embodiments of this application provide a three-dimensional image cross-modal registration device, comprising: an acquisition module for acquiring source point clouds and target point clouds of a target region; a first registration module for performing a first registration of the source point cloud and target point cloud using a sample consistency initial alignment algorithm and outputting an initial transformation matrix; and a second registration module for performing a second registration of the source point cloud and target point cloud using an improved iterative nearest point algorithm and the initial transformation matrix, until a final transformation matrix is ​​output, thereby completing the spatial alignment and fusion of the source point cloud and the target point cloud.

[0013] In one embodiment, the acquisition module is used to acquire a first modal image and a second modal image of the target area. The first modal image and the second modal image include images acquired by different imaging principles that can provide complementary anatomical structural information. The first modal image is segmented to obtain first segmentation data. The second modal image is segmented to obtain second segmentation data. A first surface triangular mesh is extracted from the first segmentation data, and a source point cloud is generated by uniformly sampling the first surface triangular mesh. A second surface triangular mesh is extracted from the second segmentation data, and a target point cloud is generated by sampling the second surface triangular mesh.

[0014] In one embodiment, the first registration module is used to extract the local geometric feature descriptors of each point in the source point cloud and the local geometric feature descriptors of the target point cloud; based on the local geometric feature descriptors of the source point cloud and the target point cloud, candidate corresponding point pairs are established; the candidate corresponding point pairs are iteratively optimized through a random sampling consistency framework and a loss function to obtain an initial transformation matrix. The initial transformation matrix represents the rigid transformation matrix that minimizes the global loss function, and the initial transformation matrix is ​​used as the first registration result.

[0015] In one embodiment, the first registration module is used to calculate the first fast point feature histogram of each point in the source point cloud and the second fast point feature histogram of each point in the target point cloud; randomly select multiple sampling points from the source point cloud, and ensure that the Euclidean distance between any two sampling points is greater than a preset threshold; use the first fast point feature histogram of each sampling point as the query basis, perform nearest neighbor search in the second fast point feature histogram space of the target point cloud, determine candidate corresponding points with similar features for each sampling point, and construct candidate corresponding point pairs.

[0016] In one embodiment, the second registration module is used to transform the source point cloud using an initial transformation matrix to obtain an initial aligned point cloud; to filter corresponding point pairs between the initial aligned point cloud and the target point cloud using normal vector angle constraints and local curvature difference constraints to obtain a set of valid corresponding point pairs; to solve for the rigid transformation increment of the valid corresponding point pair set using a weighted least squares objective function; and to update the current source point cloud using the rigid transformation increment until the convergence condition is met to obtain the target transformation matrix. The target transformation matrix includes a fine registration transformation matrix composed of all iterative transformation increments, and the fine registration transformation matrix represents the second registration result.

[0017] In one embodiment, in the normal vector angle constraint, the unit normal vector angle of the candidate corresponding point pair is less than a preset angle threshold, and in the local curvature difference constraint, the normalized local curvature absolute value of the candidate corresponding point pair is less than a preset curvature threshold.

[0018] In one embodiment, the acquisition module is further configured to calculate the loss value by half the square of the error when the error between corresponding point pairs after transformation is less than a preset error threshold; and to obtain the loss value by multiplying the preset error threshold by the absolute value of the error and then subtracting half the square of the preset threshold when the error between corresponding point pairs after transformation is greater than or equal to the preset error threshold.

[0019] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the first aspect and any possible implementation method.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of the first aspect and any possible implementation.

[0021] The beneficial effects of this application are as follows: First, this application segments the target area with high precision, transforming the multimodal registration problem into a geometric alignment problem of the same anatomical entity; then, it adopts a two-stage point cloud registration strategy from coarse to fine, first obtaining a robust initial transformation through an initial alignment algorithm based on sampling consistency, and then performing fine registration by introducing an improved iterative nearest point algorithm, ultimately achieving sub-millimeter-level multimodal image fusion, laying a solid foundation for precise orthopedic surgical navigation. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a three-dimensional image cross-modal registration method in one embodiment; Figure 2 One embodiment illustrates a cross-modal registration system architecture; Figure 3 This is a structural block diagram of a three-dimensional image cross-modal registration device shown in one embodiment; Figure 4 This is an internal structural diagram of a computer device shown in one embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0025] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0026] To facilitate understanding of the technical solutions provided in the embodiments of this application, the design concept of the embodiments of this application will be introduced first below: In precision orthopedic surgeries (such as pedicle screw implantation and pelvic fracture reduction), achieving millimeter-level surgical manipulations presents a significant challenge due to the complex bony structures, narrow surgical fields, and the requirements of minimally invasive techniques. Image-guided surgical navigation systems, by aligning the patient's real-time anatomical structures with preoperative images with high precision, provide surgeons with "fluoroscopic" capabilities beyond direct vision, making them crucial for improving surgical safety and accuracy.

[0027] In this context, multimodal image fusion demonstrates indispensable value: computed tomography (CT) can clearly present bony anatomical details, while magnetic resonance imaging (MRI) can excellently display soft tissues such as nerves and blood vessels. Precise spatial registration and fusion of the two can provide a composite three-dimensional map of both bones and soft tissues for surgical planning and navigation, achieving information complementarity.

[0028] Cross-modal 3D image registration is the core link in this technology chain, and its accuracy and robustness directly determine the reliability of the navigation system. Traditional registration methods are mostly based on iterative optimization of image grayscale or features, solving spatial transformation by maximizing a certain similarity metric. However, these methods often suffer from slow convergence speed and long computation time, making it difficult to meet the clinical needs of real-time registration in the operating room.

[0029] In recent years, deep learning-based methods have brought a new paradigm to image registration. These methods leverage the powerful feature learning capabilities of convolutional neural networks (CNNs) to achieve fast, end-to-end registration, significantly improving computational efficiency. However, existing methods still have limitations: firstly, the performance of many learning-based registration networks heavily relies on large-scale, high-quality labeled data, limiting their generalization ability in the medical field where labeled data is scarce; secondly, at the implementation level, widely adopted algorithms such as iterative nearest point and their variants are extremely sensitive to the initial pose of point cloud data and are easily plagued by local optima, posing a particularly significant challenge in CT and MRI point cloud registration where modal differences are large and overlapping areas may be incomplete.

[0030] To address the above problems, this invention proposes a novel lumbar spine CT-MRI registration scheme. The core concept of this scheme is as follows: First, high-precision segmentation of the vertebral structure is achieved using an advanced Segment Anything Model (SAM model), transforming the multimodal registration problem into a geometric alignment problem of the same anatomical entity. Then, a two-stage point cloud registration strategy, from coarse to fine, is employed. First, a robust initial transformation is obtained through a sample consistency-based initial alignment algorithm (SAC-IA algorithm). Then, a refined registration is performed using an improved iterative nearest point algorithm (ICP algorithm) incorporating multiple geometric constraints. Finally, sub-millimeter-level multimodal image fusion is achieved, laying a solid foundation for precise orthopedic surgical navigation.

[0031] This application uses medical imaging as an example to provide a detailed description of a three-dimensional image cross-modal registration method proposed in this application: In one embodiment, such as Figure 1 This paper presents a method for cross-modal registration of three-dimensional images, including the following steps: Step S101: Obtain the source point cloud and target point cloud of the target area.

[0032] In some possible embodiments, acquiring source point clouds and target point clouds of the target region includes: acquiring a first modal image and a second modal image of the target region, wherein the first modal image and the second modal image include images acquired by different imaging principles that can provide complementary anatomical structural information; performing target structure segmentation on the first modal image to obtain first segmentation data; performing target structure segmentation on the second modal image to obtain second segmentation data; extracting a first surface triangular mesh from the first segmentation data and uniformly sampling the first surface triangular mesh to generate a source point cloud; extracting a second surface triangular mesh from the second segmentation data and sampling the second surface triangular mesh to generate a target point cloud.

[0033] In some possible embodiments, the source point cloud may be generated based on a first modal image or a second modal image, and the target point cloud may be generated based on a first modal image or a second modal image. The first modal image may be a CT image or an MRI image, and the second modal image may be a CT image or an MRI image.

[0034] In some possible embodiments, the target site includes the target lumbar spine. First, CT and MRI images of the human body are obtained preoperatively through tomography and magnetic resonance imaging. The CT images include voxel information, spatial resolution, and image location information in physical space.

[0035] CT images refer to two-dimensional tomographic or three-dimensional reconstructed images of the internal anatomical structures of the human body obtained through computed tomography (CT) scanning technology. CT images can clearly display tissue structures with significant density differences, such as bones, and are used in orthopedic surgical navigation to present bony anatomical details such as vertebrae and pelvis. MRI images refer to two-dimensional tomographic or three-dimensional reconstructed images of the internal anatomical structures of the human body obtained through magnetic resonance imaging (MRI) technology. MRI images provide excellent soft tissue contrast, clearly displaying soft tissue structures such as nerves, blood vessels, intervertebral discs, and muscles, and are used in orthopedic surgical navigation to present important soft tissue anatomical information adjacent to bony structures.

[0036] Therefore, CT and MRI images have different imaging principles and tissue display characteristics, and they complement each other in clinical applications: CT images display bony structures excellently, but have poor contrast in soft tissues; MRI images display soft tissues clearly, but their display of bony structures is not as good as CT. Cross-modal registration and fusion of CT and MRI images can provide a composite three-dimensional anatomical map of both bones and soft tissues for surgical planning and navigation, achieving information complementarity and improving the safety and accuracy of surgery.

[0037] After acquiring CT and MRI images, preprocessing is performed on them. If the target area includes the lumbar spine, the preprocessing includes: First, the segmentation data of the lumbar spine is separated, and the preoperative CT and MRI data are resampled in all directions to unify the voxel spatial resolution. Then, the preoperative CT and MRI data are segmented using the SAM model to obtain CT lumbar spine segmentation data and MRI lumbar spine segmentation data.

[0038] Secondly, point cloud extraction was performed on the CT and MRI lumbar spine segmentation data. Morphological post-processing (such as hole filling and smoothing) was applied to the 3D vertebral 2D masks obtained from CT and MRI segmentation, and their surface triangular network models were extracted. Subsequently, the mesh surface was uniformly sampled to generate a 3D point cloud set for registration, denoted as the source point cloud. and target point cloud .

[0039] To overcome the problem of large initial spatial differences between multimodal images and the tendency of direct registration to get trapped in local optima, this invention first adopts a point cloud coarse registration strategy based on the Sample Consistency Initial Alignment (SAC-IA) algorithm. This aims to provide a globally approximate correct initial transformation estimate for subsequent fine registration, ensuring the robustness and convergence of the registration process.

[0040] Step S102: Perform the first registration of the source point cloud and the target point cloud using the sample consistency initial alignment algorithm, and output the initial transformation matrix.

[0041] In some possible embodiments, the source point cloud and the target point cloud are first registered using a sample consistency initial alignment algorithm, including: extracting local geometric feature descriptors for each point in the source point cloud and the local geometric feature descriptors for the target point cloud; establishing candidate corresponding point pairs based on the local geometric feature descriptors of the source point cloud and the target point cloud; iteratively optimizing the candidate corresponding point pairs using a random sample consistency framework and a loss function to obtain an initial transformation matrix, which represents the rigid transformation matrix that minimizes the global loss function, and using the initial transformation matrix as the first registration result.

[0042] In some possible embodiments, establishing candidate corresponding point pairs includes: calculating a first fast point feature histogram for each point in the source point cloud and a second fast point feature histogram for each point in the target point cloud; randomly selecting multiple sampling points from the source point cloud, and ensuring that the Euclidean distance between any two sampling points is greater than a preset threshold; using the first fast point feature histogram of each sampling point as the query basis, performing a nearest neighbor search in the second fast point feature histogram space of the target point cloud, determining candidate corresponding points with similar features for each sampling point, and constructing candidate corresponding point pairs.

[0043] As a specific example, the specific steps of the first registration (coarse registration) include: 1) Calculate the Fast Point Feature Histogram (FPFH) for each point in the source point cloud P and the target point cloud Q. As a local geometric feature robust to point cloud density and noise, FPFH can effectively characterize the complex curvature and normal distribution of the lumbar vertebral body surface, laying the foundation for subsequent corresponding point search.

[0044] 2) In the source point cloud P, randomly select a set of sampling points (number of sampling points ≥ 3). To ensure the representativeness and spatial distribution of the sampling, the algorithm constrains that the Euclidean distance between any two sampling points must be greater than a preset threshold, thereby ensuring that the selected sampling points cover different local geometric features and avoiding feature redundancy.

[0045] 3) For each source sampling point selected in the previous step, using its FPFH feature vector as the query basis, a fast nearest neighbor search is performed in the target point cloud using a KD-tree (K-Dimensional Tree) to find n candidate points with the most similar FPFH features. One of these n candidate points is randomly selected and established as the feature-corresponding point pair. This process establishes one or more possible correspondences for each source sampling point.

[0046] 4) Based on the candidate corresponding point pair set generated in the above steps, the algorithm performs iterative optimization within the Random Sample Consensus (RANSAC) framework. In each iteration: ① Randomly select the minimum required number of corresponding point pairs and calculate a candidate rigid transformation matrix. ; ② Apply this transformation to the source point cloud and evaluate the transformation quality of all corresponding point pairs using the Huber loss function. The Huber loss function is defined as follows: (1) in, The error between corresponding points in the i-th group after transformation. This is a preset threshold value. This function is more robust to outliers.

[0047] ③ Iterate through the above sampling and evaluation process, and finally return the transformation matrix that minimizes the global loss function. That is, the initial transformation matrix, which serves as the optimal output of the coarse registration.

[0048] The ultimate goal of coarse registration is to find the optimal rigid transformation that minimizes the alignment error of most correct point pairs from among the candidate corresponding points that are most likely to contain erroneous matches through the above iterative process. This transformation is the high-quality initial transformation matrix required for the subsequent fine registration stage.

[0049] Based on the initial pose obtained from coarse registration, this invention employs an improved Iterative Closest Point (ICP) algorithm for fine registration, aiming to achieve high-precision spatial alignment at the sub-voxel level. This method effectively improves the accuracy and robustness of registration by introducing a rigorous corresponding point selection mechanism and a weighted optimization strategy.

[0050] Step S103: The source point cloud and the target point cloud are registered a second time using an improved iterative nearest point algorithm and an initial transformation matrix until the final transformation matrix is ​​output, so as to complete the spatial alignment and fusion of the source point cloud and the target point cloud.

[0051] In some possible embodiments, a second registration of the source point cloud and the target point cloud is performed based on the initial transformation matrix, including: transforming the source point cloud using the initial transformation matrix to obtain an initial aligned point cloud; filtering corresponding point pairs between the initial aligned point cloud and the target point cloud using normal vector angle constraints and local curvature difference constraints to obtain a set of valid corresponding point pairs; solving for the rigid transformation increment on the set of valid corresponding point pairs using a weighted least squares objective function; updating the current source point cloud using the rigid transformation increment until the convergence condition is met to obtain the target transformation matrix, which includes a fine registration transformation matrix composed of all iterative transformation increments, and the fine registration transformation matrix represents the result of the second registration.

[0052] In some possible embodiments, in the normal vector angle constraint, the unit normal vector angle of the candidate corresponding point pair is less than a preset angle threshold, and in the local curvature difference constraint, the normalized local curvature absolute value of the candidate corresponding point pair is less than a preset curvature threshold.

[0053] As a specific example, the specific steps of the second registration (fine registration) include: 1) The transformation matrix obtained from coarse registration Applied to the source point cloud P, an initial aligned point cloud is obtained. This serves as the starting point for fine-registration iteration.

[0054] 2) In each iteration, to build the point cloud The correspondence between the target point cloud Q and the target point cloud Q abandons the traditional ICP strategy of relying solely on the nearest neighbor search based on Euclidean distance. Instead, it introduces dual geometric constraints of normal vector consistency and local curvature similarity to construct a more reliable set of point pairs. ① For a pair of candidate points and The unit normal vector is required. and The angle between Less than the preset threshold ,Right now This constraint ensures that the matching points have similar local surface orientations; ②At the same time, the local curvature of the two points is required. and The normalized absolute value is less than a preset threshold, i.e. This constraint ensures the consistency of the local curvature at the matching points.

[0055] Only when a pair of candidate points simultaneously satisfies the conditions of closest Euclidean distance, normal vector constraint, and curvature constraint is it accepted as a valid corresponding point pair. This screening can significantly suppress erroneous matches caused by noise, resolution differences, or partially overlapping regions.

[0056] 3) The set of valid corresponding points obtained after filtering Assign a confidence weight to each pair of points. The weights can be determined based on their Euclidean distance. Calculations are performed to reduce the impact of potential outliers. Then, the optimal rigid transformation for the current iteration is solved by minimizing the weighted least squares objective function. (2) in, As the confidence level weight, For rotation matrix, Let be the translation vector, ( , ) C is the set of valid corresponding point pairs. The source point after transformation. The target point.

[0057] This optimization problem can be solved efficiently and in a closed-loop manner using the weighted SVD method to obtain the transformation increment for this iteration. .

[0058] 4) Update the current source point cloud using the solved transformation increment: (3) in, This is the source point cloud before the update. For rotation increment matrix, This is the translation increment vector. The algorithm is considered convergent when the update amount of the transformation matrix between two consecutive iterations is less than a preset threshold, or when the region of change of the objective function value is stable.

[0059] Finally, the output is a finely registered transformation matrix composed of all iterative transformation increments. This enables high-precision rigid registration from CT image space to MRI image space.

[0060] In some possible embodiments, the first registration can be a coarse registration, and the second registration can be a fine registration. Figure 2 A cross-modal registration system architecture diagram is shown, such as Figure 2As shown in the figure, first, the preoperative CT data and preoperative MRI data are respectively input into the SAM model for vertebral structure segmentation to obtain the corresponding CT segmentation data and MRI segmentation data. Point cloud extraction is performed on the CT segmentation data and MRI segmentation data respectively to generate CT point cloud data (source point cloud) and MRI point cloud data (target point cloud). The CT point cloud data and MRI point cloud data are input into the SAC-IA algorithm for initial registration: the transformation matrix is calculated through iterative optimization, and it is judged whether the current loss value is less than the preset minimum loss threshold. If the condition (loss ≤ loss_min) is satisfied, the current transformation is accepted to obtain the initial transformation matrix M_tough = (R_tough, t_tough). If the condition is not satisfied, continue iterative optimization until the optimal transformation matrix M_best = (R_best, t_best) that minimizes the loss function is found. Taking the rough registration result as the initial value, input it into the improved ICP algorithm for fine registration: double geometric constraints are introduced during the iteration process for corresponding point screening: angle constraint (θ<θ_th): it is required that the included angle between the normal vectors of the corresponding point pairs is less than the threshold, and curvature constraint (c<c_th): it is required that the local curvature difference between the corresponding point pairs is less than the threshold. The registration accuracy is gradually improved through iterative optimization. Finally, the rough registration effect and fine registration effect are output to achieve high-precision spatial alignment of CT and MRI images.

[0061] In some possible embodiments, the loss value of the loss function is calculated in the following manner: when the error between the corresponding point pairs after transformation is less than the preset error threshold, the loss value is calculated by half of the square of the error; when the error between the corresponding point pairs after transformation is greater than or equal to the preset error threshold, the loss value is obtained by multiplying the preset error threshold by the absolute value of the error and then subtracting half of the square of the preset threshold.

[0062] Before point cloud extraction, it is necessary to make a dataset for the SAM model. The dataset in the experiment is made from 100 real lumbar CTs and the corresponding MRI images. The three-dimensional CT image and the corresponding MRI image of a target lumbar vertebra are regarded as a sample. Through data augmentation, a preset number of samples are obtained, for example: 1000 samples, which are divided into a training set and a test set according to the ratio of 8:2.

[0063] Then, iterative regression training and prediction are performed on the neural network model. The overall network framework is built based on Pytorch. The Ranger optimizer is used to train the network. The initial learning rate is 1e-4, and a total of 1000 epochs are trained. It is trained on the Nvidia GTX 3090 GPU, and the batch size is 64. During the training process, each sample is subjected to a specified number of iterative trainings. The relevant parameters for ICP algorithm registration: the maximum number of iterations is 100, the maximum neighbor distance is 10 mm, and the mean square error is 0.001.

[0064] During the test, the input data consisted of preoperative 3D CT and 3D MRI data. A trained model was used for prediction, and the prediction matrix was returned when the number of iterations n reached a specified value. .

[0065] After training the neural network model, perform error evaluation on the output results: ① By predicting the pose matrix By estimating the pose change of point cloud data P, a predicted point cloud can be obtained. The mean square error between the predicted point cloud and the target point cloud is calculated as follows: (4) in, Let N be the mean square error, and N be the total number of points. For the i-th point in the target point cloud, To predict the i-th point in the point cloud.

[0066] ②The average symmetric surface distance (ASSD) between the predicted point cloud and the target point cloud Q can be expressed as: (5) in, The average symmetrical surface distance, Point cloud The number of midpoints Point The shortest distance to point cloud Q. Q is the target point cloud. To predict point clouds, where, The smaller the value, the closer the two point cloud surfaces are, and the higher the registration accuracy.

[0067] In summary, the segmentation method based on the SAM segmentation model in this application effectively improves segmentation accuracy and efficiency. The Sample Consistency-Based Initial Alignment (SAC-IA) algorithm is used to obtain the initial transformation matrix, providing a good initial value for subsequent fine registration. Furthermore, based on the initial registration matrix, an improved ICP algorithm with dual geometric constraints is introduced to achieve fine registration of multimodal images. This two-stage strategy combining SAC-IA coarse registration and improved ICP fine registration effectively avoids local optima and improves registration accuracy.

[0068] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0069] Based on the same inventive concept, embodiments of this application provide a three-dimensional image cross-modal registration device, including: an acquisition module 301, a first registration module 302, and a second registration module 303, wherein: The acquisition module 301 is used to acquire the source point cloud and the target point cloud of the target area; The first registration module 302 is used to perform the first registration of the source point cloud and the target point cloud using the sample consistency initial alignment algorithm and output the initial transformation matrix. The second registration module 303 is used to perform a second registration of the source point cloud and the target point cloud using an improved iterative nearest point algorithm and an initial transformation matrix, until the final transformation matrix is ​​output, so as to complete the spatial alignment and fusion of the source point cloud and the target point cloud.

[0070] In one embodiment, the acquisition module is used to acquire a first modal image and a second modal image of the target area. The first modal image and the second modal image include images acquired by different imaging principles that can provide complementary anatomical structural information. The first modal image is segmented to obtain first segmentation data. The second modal image is segmented to obtain second segmentation data. A first surface triangular mesh is extracted from the first segmentation data, and a source point cloud is generated by uniformly sampling the first surface triangular mesh. A second surface triangular mesh is extracted from the second segmentation data, and a target point cloud is generated by sampling the second surface triangular mesh.

[0071] In one embodiment, the first registration module is used to extract the local geometric feature descriptors of each point in the source point cloud and the local geometric feature descriptors of the target point cloud; based on the local geometric feature descriptors of the source point cloud and the target point cloud, candidate corresponding point pairs are established; the candidate corresponding point pairs are iteratively optimized through a random sampling consistency framework and a loss function to obtain an initial transformation matrix. The initial transformation matrix represents the rigid transformation matrix that minimizes the global loss function, and the initial transformation matrix is ​​used as the first registration result.

[0072] In one embodiment, the first registration module is used to calculate the first fast point feature histogram of each point in the source point cloud and the second fast point feature histogram of each point in the target point cloud; randomly select multiple sampling points from the source point cloud, and ensure that the Euclidean distance between any two sampling points is greater than a preset threshold; use the first fast point feature histogram of each sampling point as the query basis, perform nearest neighbor search in the second fast point feature histogram space of the target point cloud, determine candidate corresponding points with similar features for each sampling point, and construct candidate corresponding point pairs.

[0073] In one embodiment, the second registration module is used to transform the source point cloud using an initial transformation matrix to obtain an initial aligned point cloud; to filter corresponding point pairs between the initial aligned point cloud and the target point cloud using normal vector angle constraints and local curvature difference constraints to obtain a set of valid corresponding point pairs; to solve for the rigid transformation increment of the valid corresponding point pair set using a weighted least squares objective function; and to update the current source point cloud using the rigid transformation increment until the convergence condition is met to obtain the target transformation matrix. The target transformation matrix includes a fine registration transformation matrix composed of all iterative transformation increments, and the fine registration transformation matrix represents the second registration result.

[0074] In one embodiment, in the normal vector angle constraint, the unit normal vector angle of the candidate corresponding point pair is less than a preset angle threshold, and in the local curvature difference constraint, the normalized local curvature absolute value of the candidate corresponding point pair is less than a preset curvature threshold.

[0075] In one embodiment, the acquisition module is further configured to calculate the loss value by half the square of the error when the error between corresponding point pairs after transformation is less than a preset error threshold; and to obtain the loss value by multiplying the preset error threshold by the absolute value of the error and then subtracting half the square of the preset threshold when the error between corresponding point pairs after transformation is greater than or equal to the preset error threshold.

[0076] Specific limitations regarding the 3D image cross-modal registration device can be found in the limitations on the speech recognition method above, and will not be repeated here. Each module in the aforementioned 3D image cross-modal registration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the corresponding operations of each module.

[0077] Based on the same inventive concept, embodiments of this application provide a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a three-dimensional image cross-modal registration method.

[0078] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0079] Based on the same inventive concept, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: acquiring source point clouds and target point clouds of a target region; performing a first registration of the source point clouds and target point clouds using a sample consistency initial alignment algorithm and outputting an initial transformation matrix; performing a second registration of the source point clouds and target point clouds using an improved iterative nearest point algorithm and the initial transformation matrix, until a final transformation matrix is ​​output, thereby completing the spatial alignment and fusion of the source point clouds and target point clouds.

[0080] In one embodiment, the processor further performs the following steps when executing the computer program: Acquiring source point clouds and target point clouds of a target region includes: acquiring a first modal image and a second modal image of the target region, wherein the first modal image and the second modal image include images acquired by different imaging principles that can provide complementary anatomical structural information; performing target structure segmentation on the first modal image to obtain first segmentation data; performing target structure segmentation on the second modal image to obtain second segmentation data; extracting a first surface triangular mesh from the first segmentation data and uniformly sampling the first surface triangular mesh to generate a source point cloud; extracting a second surface triangular mesh from the second segmentation data and sampling the second surface triangular mesh to generate a target point cloud.

[0081] In one embodiment, the processor further performs the following steps when executing the computer program: The source and target point clouds are first registered using a sample consistency initial alignment algorithm, which includes: extracting local geometric feature descriptors for each point in the source point cloud and the target point cloud; establishing candidate corresponding point pairs based on the local geometric feature descriptors of the source and target point clouds; iteratively optimizing the candidate corresponding point pairs using a random sample consistency framework and loss function to obtain an initial transformation matrix. The initial transformation matrix represents the rigid transformation matrix that minimizes the global loss function, and the initial transformation matrix is ​​used as the first registration result.

[0082] In one embodiment, the processor further performs the following steps when executing the computer program: Calculate the first fast point feature histogram for each point in the source point cloud and the second fast point feature histogram for each point in the target point cloud; randomly select multiple sampling points from the source point cloud, ensuring that the Euclidean distance between any two sampling points is greater than a preset threshold; using the first fast point feature histogram of each sampling point as the query basis, perform a nearest neighbor search in the second fast point feature histogram space of the target point cloud, determine candidate corresponding points with similar features for each sampling point, and construct candidate corresponding point pairs.

[0083] In one embodiment, the processor further performs the following steps when executing the computer program: The source point cloud is transformed using an initial transformation matrix to obtain an initial aligned point cloud. Corresponding point pairs between the initial aligned point cloud and the target point cloud are then selected using normal vector angle constraints and local curvature difference constraints to obtain a set of valid corresponding point pairs. The rigid transformation increment is solved using a weighted least squares objective function on the set of valid corresponding point pairs. The current source point cloud is updated using the rigid transformation increment until the convergence condition is met, resulting in the target transformation matrix. The target transformation matrix includes a fine registration transformation matrix composed of all iterative transformation increments, which represents the second registration result.

[0084] In one embodiment, the processor further performs the following steps when executing the computer program: In the normal vector angle constraint, the unit normal vector angle of the candidate corresponding point pair is less than a preset angle threshold. In the local curvature difference constraint, the normalized local curvature absolute value of the candidate corresponding point pair is less than a preset curvature threshold.

[0085] In one embodiment, the processor further performs the following steps when executing the computer program: When the error between corresponding point pairs after transformation is less than the preset error threshold, the loss value is calculated by half the square of the error; when the error between corresponding point pairs after transformation is greater than or equal to the preset error threshold, the loss value is obtained by multiplying the preset error threshold by the absolute value of the error and then subtracting half the square of the preset threshold.

[0086] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it performs the following steps: acquiring source point clouds and target point clouds of the target region; performing a first registration of the source point clouds and target point clouds using a sample consistency initial alignment algorithm and outputting an initial transformation matrix; performing a second registration of the source point clouds and target point clouds using an improved iterative nearest point algorithm and the initial transformation matrix, until the final transformation matrix is ​​output, so as to complete the spatial alignment and fusion of the source point clouds and target point clouds.

[0087] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: acquiring source point clouds and target point clouds of the target region, including: acquiring a first modal image and a second modal image of the target region, wherein the first modal image and the second modal image include images acquired by different imaging principles that can provide complementary anatomical structural information; performing target structure segmentation on the first modal image to obtain first segmentation data; performing target structure segmentation on the second modal image to obtain second segmentation data; extracting a first surface triangular mesh from the first segmentation data and uniformly sampling the first surface triangular mesh to generate a source point cloud; extracting a second surface triangular mesh from the second segmentation data and sampling the second surface triangular mesh to generate a target point cloud.

[0088] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: performing a first registration of the source point cloud and the target point cloud using a sample consistency initial alignment algorithm, including: extracting local geometric feature descriptors for each point in the source point cloud and local geometric feature descriptors for the target point cloud; establishing candidate corresponding point pairs based on the local geometric feature descriptors of the source point cloud and the target point cloud; iteratively optimizing the candidate corresponding point pairs using a random sample consistency framework and a loss function to obtain an initial transformation matrix, wherein the initial transformation matrix represents the rigid transformation matrix that minimizes the global loss function, and the initial transformation matrix is ​​used as the first registration result.

[0089] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: calculating a first fast point feature histogram for each point in the source point cloud and a second fast point feature histogram for each point in the target point cloud; randomly selecting multiple sampling points from the source point cloud, and ensuring that the Euclidean distance between any two sampling points is greater than a preset threshold; using the first fast point feature histogram of each sampling point as the query basis, performing a nearest neighbor search in the second fast point feature histogram space of the target point cloud, determining candidate corresponding points with similar features for each sampling point, and constructing candidate corresponding point pairs.

[0090] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: transforming the source point cloud using an initial transformation matrix to obtain an initial aligned point cloud; filtering corresponding point pairs between the initial aligned point cloud and the target point cloud using normal vector angle constraints and local curvature difference constraints to obtain a set of valid corresponding point pairs; solving for the rigid transformation increment on the set of valid corresponding point pairs using a weighted least squares objective function; updating the current source point cloud using the rigid transformation increment until the convergence condition is met to obtain the target transformation matrix, which includes a fine registration transformation matrix composed of all iterative transformation increments, and the fine registration transformation matrix represents the second registration result.

[0091] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: in the normal vector angle constraint, the unit normal vector angle of the candidate corresponding point pair is less than a preset angle threshold, and in the local curvature difference constraint, the normalized local curvature absolute value of the candidate corresponding point pair is less than a preset curvature threshold.

[0092] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: when the error between corresponding point pairs after transformation is less than a preset error threshold, the loss value is calculated by half the square of the error; when the error between corresponding point pairs after transformation is greater than or equal to the preset error threshold, the loss value is obtained by multiplying the preset error threshold by the absolute value of the error and then subtracting half the square of the preset threshold.

[0093] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0094] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for cross-modal registration of three-dimensional images, characterized in that, include: Obtain the source point cloud and target point cloud of the target area; The source point cloud and the target point cloud are first registered using the sample consistency initial alignment algorithm, and the initial transformation matrix is ​​output. The source point cloud and the target point cloud are registered a second time using an improved iterative nearest-point algorithm and the initial transformation matrix until the final transformation matrix is ​​output, thereby completing the spatial alignment and fusion of the source point cloud and the target point cloud.

2. The method according to claim 1, characterized in that, Obtain the source point cloud and target point cloud of the target area, including: Acquire a first modal image and a second modal image of the target area, wherein the first modal image and the second modal image include images acquired by different imaging principles that can provide complementary anatomical structural information; The first modality image is segmented into its target structure to obtain first segmentation data; the second modality image is segmented into its target structure to obtain second segmentation data. The first surface triangular mesh is extracted using the first segmentation data, and the first surface triangular mesh is uniformly sampled to generate a source point cloud; the second surface triangular mesh is extracted using the second segmentation data, and the target point cloud is generated by sampling the second surface triangular mesh.

3. The method according to claim 1, characterized in that, The source point cloud and the target point cloud are first registered using a sample consistency initial alignment algorithm, including: Extract the local geometric feature descriptor of each point in the source point cloud and the local geometric feature descriptor of the target point cloud; Candidate corresponding point pairs are established based on the local geometric feature descriptors of the source point cloud and the target point cloud. The candidate corresponding point pairs are iteratively optimized using a random sampling consistency framework and a loss function to obtain an initial transformation matrix. The initial transformation matrix represents the rigid transformation matrix that minimizes the global loss function, and the initial transformation matrix is ​​used as the first registration result.

4. The method according to claim 3, characterized in that, Establish candidate corresponding point pairs, including: Calculate the first fast point feature histogram for each point in the source point cloud and the second fast point feature histogram for each point in the target point cloud; Multiple sampling points are randomly selected from the source point cloud, and the Euclidean distance between any two sampling points is greater than a preset threshold. Using the first fast point feature histogram of each sampling point as the query basis, a nearest neighbor search is performed in the second fast point feature histogram space of the target point cloud to determine candidate corresponding points with similar features for each sampling point and construct candidate corresponding point pairs.

5. The method according to claim 1, characterized in that, A second registration is performed on the source point cloud and the target point cloud based on the initial transformation matrix, including: The source point cloud is transformed using the initial transformation matrix to obtain an initial aligned point cloud; By using normal vector angle constraints and local curvature difference constraints, the initial aligned point cloud and the target point cloud are filtered for corresponding point pairs to obtain a set of valid corresponding point pairs. The rigid transformation increment is solved for the set of effective corresponding point pairs by using a weighted least squares objective function; The current source point cloud is updated by the rigid transformation increment until the convergence condition is met, and the target transformation matrix is ​​obtained. The target transformation matrix includes a fine registration transformation matrix composed of all iterative transformation increments, and the fine registration transformation matrix represents the second registration result.

6. The method according to claim 5, characterized in that, In the normal vector angle constraint, the unit normal vector angle of the candidate corresponding point pair is less than a preset angle threshold, and in the local curvature difference constraint, the normalized local curvature absolute value of the candidate corresponding point pair is less than a preset curvature threshold.

7. The method according to claim 3, characterized in that, The loss value of the loss function is calculated in the following way: When the error between corresponding point pairs after transformation is less than the preset error threshold, the loss value is calculated by half the square of the error. When the error between corresponding point pairs after transformation is greater than or equal to a preset error threshold, the loss value is obtained by multiplying the preset error threshold by the absolute value of the error and then subtracting half of the square of the preset threshold.

8. A three-dimensional image cross-modal registration device, characterized in that, include: The acquisition module is used to acquire the source point cloud and the target point cloud of the target area; The first registration module is used to perform the first registration of the source point cloud and the target point cloud using the sample consistency initial alignment algorithm, and output the initial transformation matrix. The second registration module is used to perform a second registration of the source point cloud and the target point cloud using an improved iterative nearest point algorithm and the initial transformation matrix, until the final transformation matrix is ​​output, so as to complete the spatial alignment and fusion of the source point cloud and the target point cloud.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the three-dimensional image cross-modal registration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor and implementing the three-dimensional image cross-modal registration method as described in any one of claims 1 to 7.

Images