Universal multimodal neural image registration method, device and readable medium
By performing 3D rendering and planar correction of image data in 3DSlicer software, combined with coordinate system transformation and manual fine-tuning, the inefficiency of existing multimodal neural image registration algorithms under low-quality images is solved, realizing an efficient and universal image registration method.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAMEN UNIV
- Filing Date
- 2024-02-02
- Publication Date
- 2026-07-17
AI Technical Summary
Existing multimodal neural image registration algorithms consume a lot of computational resources when processing large-scale images or complex deformations, and have high requirements for image quality, resulting in low efficiency. They are particularly prone to failure when the image scanning quality is poor or the contrast is low.
3DSlicer software was used for 3D rendering and planar correction of image data. A coordinate system was established by selecting three points on the corrected image plane, the transformation matrix was calculated for pre-registration, and manual fine-tuning was combined to complete the registration, including single-axis rotation and bidirectional translation.
It achieves universal registration unaffected by image scan quality and contrast, reduces the requirements for image quality, improves processing efficiency, avoids automatic registration failures caused by incomplete image scans or poor resolution, and reduces the economic burden on patients.
Smart Images
Figure CN118115553B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing, and specifically to a universal multimodal neural image registration method, apparatus, and readable medium. Background Technology
[0002] The diagnosis of neurological diseases such as traumatic brain injury, cerebral hemorrhage, and brain tumors relies heavily on neuroimaging, such as brain CT scans and magnetic resonance imaging (MRI). These images are obtained by scanning with different medical imaging devices, a process known as multimodal imaging. Each imaging modality provides different information about the patient's brain lesions; for example, a brain CT scan can show information about cerebral hemorrhage and the skull, while a brain MRI can show information about brain tumors.
[0003] Multimodal image registration is a crucial technology in the field of medical image processing. Its purpose is to spatially align medical images of different modalities to leverage their respective advantages, enabling doctors to make more accurate disease diagnoses, surgical planning, and other applications.
[0004] Existing registration algorithms include:
[0005] 1. Feature point matching: This method automatically detects feature points in an image and then searches for similar feature points in different modal images for matching. The drawback is that feature point matching is prone to failure in cases of low contrast or poor image quality.
[0006] 2. Least Squares Method: Select at least three sets of corresponding points on both modal images, such as the tip of the nose, the root of the nose, and the tragus. Then, calculate the deformation matrix of the three sets of points using the least squares method, and apply it to the original CT or MRI scan. The drawback is that when the slice thickness of the image scan is large, inaccurate point selection often occurs, leading to significant registration errors.
[0007] Therefore, existing algorithms require a lot of computational resources when processing large-scale images or complex deformations, which affects efficiency. Moreover, existing methods have high requirements for image quality. Summary of the Invention
[0008] In view of the aforementioned deficiencies in the existing technology for multimodal neural image registration, the purpose of the embodiments of this application is to propose a universal multimodal neural image registration method, apparatus, and readable medium to solve the technical problems mentioned in the background section. This method is unaffected by image scanning quality and contrast and can register any multimodal image.
[0009] In a first aspect, the present invention provides a general multimodal neural image registration method, comprising the following steps:
[0010] Acquire multimodal image data and import them into medical image processing software. The multimodal image data includes brain CT image data and brain magnetic resonance MRI image data stored in DICOM format.
[0011] Each data point is assigned a moving data point corresponding to one of the multimodal image data points and a stationary data point corresponding to the other. Planar correction is then performed on the moving data and the stationary data to obtain the registration planes of the corrected moving data and the corrected stationary data.
[0012] On the registration planes of the corrected moving data and the corrected stationary data, select any three points on non-straight lines. Establish moving coordinate systems and stationary coordinate systems based on the selected three points. Calculate the transformation matrix from the moving coordinate system to the stationary coordinate system. Use the transformation matrix to transform the moving data to obtain the transformed moving data. Use the transformed moving data and stationary data to achieve pre-registration of the registration plane.
[0013] Fine-tuning is performed on the transformed motion data to obtain the registration results of the multimodal image data.
[0014] Preferably, medical image processing software includes 3DSlicer software, and also includes:
[0015] After importing the multimodal image data into the 3DSlicer software, the rendering extension module in the 3DSlicer software is used to perform 3D rendering on the multimodal image data to obtain the 3D rendered multimodal image data. The moving data is one of the 3D rendered multimodal image data, and the stationary data is the other of the 3D rendered multimodal image data.
[0016] As a preferred method, the crosshair positioning function in the 3DSlicer software is used for plane correction.
[0017] As a preferred method, any three points can be selected using the mark extension module in the 3DSlicer software.
[0018] Preferably, a moving coordinate system and a stationary coordinate system are established based on the three selected points, and the transformation matrix from the moving coordinate system to the stationary coordinate system is calculated, specifically including:
[0019] The coordinates of three points, t1, t2, and t3, are obtained from the registration plane of the corrected motion data and the registration plane of the corrected stationary data in the medical image processing software. The specific calculation formula is as follows:
[0020] Calculate the unit vector along the z-axis:
[0021]
[0022] Calculate the unit vector along the x-axis:
[0023]
[0024] Calculate the unit vector along the y-axis:
[0025] y = z × x;
[0026] Where, ||·|| represents the norm of a vector or matrix;
[0027] Based on the unit vectors of the x-axis, y-axis, and z-axis of the registration plane of the corrected moving data and the registration plane of the corrected stationary data, respectively establish the moving coordinate system Move and the stationary coordinate system Fix;
[0028] The transformation matrix Trans is calculated using the following code:
[0029] Move0 = Move -1 ;
[0030] Trans = Move0 Fix.
[0031] Preferably, the registration plane includes the sagittal plane, the coronal plane, or the transverse plane.
[0032] Preferably, the fine-tuning includes single-axis rotation, bidirectional vertical translation, and bidirectional forward and backward translation. The axis of the single-axis rotation is perpendicular to the registration plane, and the directions of the bidirectional vertical translation and the bidirectional forward and backward translation are perpendicular to each other and located on the registration plane.
[0033] Secondly, the present invention provides a universal multimodal neural image registration device, comprising:
[0034] The import module is configured to acquire multimodal image data and import it into medical image processing software. The multimodal image data includes brain CT image data and brain magnetic resonance MRI image data stored in DICOM format.
[0035] The correction module is configured to set the data corresponding to one of the multimodal image data as moving data and the data corresponding to the other of the multimodal image data as stationary data, and to perform planar correction on the moving data and the stationary data respectively to obtain the registration plane of the corrected moving data and the registration plane of the corrected stationary data.
[0036] The pre-registration module is configured to select any three points on non-straight lines on the registration plane of the corrected moving data and the registration plane of the corrected stationary data, respectively, establish a moving coordinate system and a stationary coordinate system based on the selected three points, calculate the transformation matrix from the moving coordinate system to the stationary coordinate system, transform the moving data using the transformation matrix to obtain the transformed moving data, and achieve pre-registration of the registration plane using the transformed moving data and stationary data.
[0037] The fine-tuning module is configured to fine-tune the transformed motion data to obtain the registration results of the multimodal image data.
[0038] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.
[0039] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] (1) The universal multimodal neural image registration method proposed in this invention is not affected by the image scanning quality and contrast, and can register any multimodal image, with strong versatility.
[0042] (2) The general multimodal neural image registration method proposed in this invention is an improvement on the existing medical image processing software. It does not have high requirements for image quality, does not require a lot of computing resources, and has high processing efficiency.
[0043] (3) The general multimodal neural image registration method proposed in this invention performs pre-registration based on the registration plane and then performs manual fine-tuning to complete rotation and translation. This solves the problem that fully automatic registration cannot be completed when the image data resolution is poor. It avoids the situation where automatic registration fails due to incomplete image scanning or poor resolution, thus requiring the re-capture of image data and reducing the economic burden on patients. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0045] Figure 1 This is a schematic flowchart illustrating a general multimodal neural image registration method according to an embodiment of this application.
[0046] Figure 2 The image shows a 3D rendering result of brain CT and MRI image data of the general multimodal neuroimage registration method according to an embodiment of this application.
[0047] Figure 3 This is a schematic diagram of the registration plane of the corrected magnetic resonance MRI image data in the general multimodal neural image registration method of the embodiments of this application;
[0048] Figure 4 This is a schematic diagram of a general multimodal neural image registration method based on points placed in the corrected sagittal plane, which is an embodiment of this application.
[0049] Figure 5 The image shows the effect of pre-registration in the sagittal plane of magnetic resonance MRI and CT, which are embodiments of the general multimodal neuroimage registration method of this application.
[0050] Figure 6 This is a diagram showing the effect of the general multimodal neural image registration method according to an embodiment of this application, where red, green and blue represent handles;
[0051] Figure 7 This is a schematic diagram of a general-purpose multimodal neural image registration device according to an embodiment of this application. Detailed Implementation
[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0053] Figure 1 This application illustrates a general multimodal neural image registration method provided by an embodiment of the present application, comprising the following steps:
[0054] S1. Acquire multimodal image data and import them into medical image processing software. The multimodal image data includes brain CT image data and brain magnetic resonance MRI image data stored in DICOM format.
[0055] In a specific embodiment, the medical image processing software includes 3DSlicer software, and further includes:
[0056] After importing the multimodal image data into the 3DSlicer software, the rendering extension module in the 3DSlicer software is used to perform 3D rendering on the multimodal image data to obtain the 3D rendered multimodal image data. The moving data is one of the 3D rendered multimodal image data, and the stationary data is the other of the 3D rendered multimodal image data.
[0057] Specifically, the patient's brain CT or MRI images are in DICOM format, a format specifically designed for medical image storage. Before performing multimodal image registration, the patient's DICOM-formatted brain CT and MRI images need to be acquired separately; this data is provided by the hospital's radiology department.
[0058] The embodiments of this application are based on the open-source medical image processing software 3DSlicer. Other related image processing software can also be used, but the usage methods are slightly different. To make the registration process clearer, the embodiments of this application use the Python programming language to develop a separate SagReg (sagittal registration) extension module based on 3DSlicer and integrate it into the 3DSlicer software for functional demonstration. Alternatively, it can be implemented using other programming languages, and it is not necessary to integrate the SagReg (sagittal registration) extension module into the 3DSlicer software; the built-in functions of the 3DSlicer software can also be used.
[0059] refer to Figure 2 The acquired brain CT and MRI images were imported into the 3DSlicer software. Then, the Volume Rendering module in the 3DSlicer software was opened to render the brain CT and MRI images in 3D. The spatial positional differences between the two modalities were observed, indicating that the two images did not overlap after 3D rendering.
[0060] S2, set the data corresponding to one of the multimodal image data as moving data and the data corresponding to the other of the multimodal image data as stationary data, respectively, and perform planar correction on the moving data and the stationary data to obtain the registration plane of the corrected moving data and the registration plane of the corrected stationary data.
[0061] In a specific embodiment, the plane correction uses the crosshair positioning function in the 3DSlicer software.
[0062] In specific embodiments, the registration plane includes the sagittal plane, the coronal plane, or the transverse plane.
[0063] Specifically, select the SagReg extension module and set the parameters as follows: Moving data refers to 3D-rendered MRI brain images, and stationary data refers to 3D-rendered CT brain images. The two modalities can be set arbitrarily, without regard to their order. Stationary data refers to data whose position does not need to change during the entire registration process; it can be either 3D-rendered CT or MRI brain images. Moving data refers to data that needs to be moved to the stationary data position according to the registration algorithm during the registration process.
[0064] This application uses the sagittal plane as the registration plane for illustrative purposes. The human brain is divided into left and right hemispheres, separated by an imaginary midline plane called the sagittal plane. Because each patient's position varies slightly during image scanning, the default sagittal plane after scanning is not standard. Therefore, it needs to be corrected using the crosshair positioning function (MPR) of the 3DSlicer software. The corrected registration planes for moving data and stationary data are shown below. Figure 3 As shown. The sagittal plane is a plane in the left-right direction. Other planes include the coronal plane (a plane in the front-back direction) and the transverse plane (a plane in the top-bottom direction). Any registration method based on the sagittal, coronal, or transverse planes should fall within the scope of protection of this patent.
[0065] S3. Select any three points on different straight lines on the registration plane of the corrected moving data and the registration plane of the corrected stationary data respectively. Establish moving coordinate system and stationary coordinate system based on the selected three points respectively. Calculate the transformation matrix from moving coordinate system to stationary coordinate system. Use the transformation matrix to transform the moving data to obtain the transformed moving data. Achieve pre-registration of the registration plane by using the transformed moving data and stationary data.
[0066] In a specific embodiment, the marker extension module in the 3DSlicer software is used to select any three points.
[0067] In a specific embodiment, a moving coordinate system and a stationary coordinate system are established based on the three selected points, and the transformation matrix from the moving coordinate system to the stationary coordinate system is calculated, specifically including:
[0068] The coordinates of three points, t1, t2, and t3, are obtained from the registration plane of the corrected motion data and the registration plane of the corrected stationary data in the medical image processing software. The specific calculation formula is as follows:
[0069] Calculate the unit vector along the z-axis:
[0070]
[0071] Calculate the unit vector along the x-axis:
[0072]
[0073] Calculate the unit vector along the y-axis:
[0074] y = z × x;
[0075] Where, ||·|| represents the norm of a vector or matrix;
[0076] Based on the unit vectors of the x-axis, y-axis, and z-axis of the registration plane of the corrected moving data and the registration plane of the corrected stationary data, respectively establish the moving coordinate system Move and the stationary coordinate system Fix;
[0077] The transformation matrix Trans is calculated using the following code:
[0078] Move0 = Move -1 ;
[0079] Trans = Move0 Fix.
[0080] For details, please refer to Figure 4 Using the Markups module of 3DSlicer software, three points not on the same straight line are arbitrarily selected on both the corrected MRI sagittal plane and the CT sagittal plane. The coordinates of these three points on the corrected MRI sagittal plane are set as m1, m2, and m3; and the coordinates of these three points on the corrected CT sagittal plane are set as c1, c2, and c3. It should be noted that the coordinates of these three points are generated by 3DSlicer software using the information inherent in the medical image as the origin. When marking on the image, these coordinates are generated based on the default origin of the medical image. If two or more sets of images are imported into 3DSlicer software simultaneously, 3DSlicer software will default to using the origin of the first set of data as the overall origin. Coordinate systems Move and Fix are established based on the selected points. The calculation process in this embodiment is completed using Python's NumPy library functions, which are integrated into the SagReg extension module. In actual operation, the SagReg extension module automatically completes the calculation. Taking the calculation of coordinate system Move as an example, the specific calculation process is as follows:
[0081] Calculate the unit vector along the z-axis:
[0082]
[0083] Calculate the unit vector along the x-axis:
[0084]
[0085] Calculate the unit vector along the y-axis:
[0086] y = z × x;
[0087] The coordinate system Move can be obtained from the above formula, and the coordinate system Fix can be obtained similarly.
[0088] Further reference Figure 5 After obtaining the coordinate system Move for the brain MRI image data and the coordinate system Fix for the brain CT image data, the next step is to calculate the transformation matrix Trans from the coordinate system Move to the coordinate system Fix. This can be done using NumPy library functions, and the code is as follows:
[0089] Move0 = Move -1 ;
[0090] Trans = Move0 Fix;
[0091] In practical applications, the above steps can be directly accomplished using NumPy library functions. The resulting transformation matrix Trans is then applied to the MRI image data to align the sagittal planes of the two modalities. The above steps employ planar registration; in other embodiments, point registration can also be used. This involves selecting corresponding points on both modalities, such as A, B, C and a, b, c, respectively, calculating the transformation matrix between the two sets of points, and applying this transformation matrix to the moving data to complete the registration.
[0092] S4, fine-tuning the transformed motion data to obtain the registration result of the multimodal image data.
[0093] In a specific embodiment, fine-tuning includes single-axis rotation, bidirectional vertical translation, and bidirectional forward and backward translation. The axis of single-axis rotation is perpendicular to the registration plane, and the directions of bidirectional vertical translation and bidirectional forward and backward translation are perpendicular to each other and located on the registration plane.
[0094] Specifically, the above process completes the pre-registration of the sagittal plane, resulting in an overlap between the sagittal planes of the MRI and CT images. Currently, the two image modalities only overlap in the sagittal plane; registration is not yet complete. Next, by using the handles displayed in the 3DSlicer software to rotate and translate the MRI and CT images, final registration with the CT images can be achieved. Fine-tuning can be performed arbitrarily until a satisfactory result is obtained. Figure 6 As shown. This completes the registration process from MRI image data to CT image data.
[0095] The key point of this invention is that it first pre-registers the standard sagittal plane of multimodal image data, restricting the multi-axis displacement of the data. That is, after pre-registration, the data can only be rotated along a single axis (perpendicular to the registration plane) and translated bidirectionally (along the vertical and horizontal directions and the forward and backward directions of the registration plane). Unlike fully automatic registration, this registration method requires the user to complete the final rotation and translation steps based on sagittal plane registration. The introduction of this method solves the problem that fully automatic registration cannot be completed when the data resolution is poor.
[0096] The steps S1-S4 above do not necessarily represent the order of the steps, but are represented by step symbols. The order of the steps can be adjusted.
[0097] Further reference Figure 7 As an implementation of the methods shown in the above figures, this application provides an embodiment of a general-purpose multimodal neural image registration device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.
[0098] This application provides a general-purpose multimodal neural image registration device, including:
[0099] Import module 1 is configured to acquire multimodal image data and import it into medical image processing software. The multimodal image data includes brain CT image data and brain magnetic resonance MRI image data stored in DICOM format.
[0100] The correction module 2 is configured to set the data corresponding to one of the multimodal image data as moving data and the data corresponding to the other of the multimodal image data as stationary data, respectively, and perform planar correction on the moving data and the stationary data to obtain the registration plane of the corrected moving data and the registration plane of the corrected stationary data.
[0101] The pre-registration module 3 is configured to select any three points on non-straight lines on the registration plane of the corrected moving data and the registration plane of the corrected stationary data, respectively, establish a moving coordinate system and a stationary coordinate system based on the selected three points, calculate the transformation matrix from the moving coordinate system to the stationary coordinate system, transform the moving data using the transformation matrix to obtain the transformed moving data, and achieve pre-registration of the registration plane by using the transformed moving data and stationary data.
[0102] Fine-tuning module 4 is configured to fine-tune the transformed motion data to obtain the registration result of multimodal image data.
[0103] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.
Claims
1. A general multimodal neural image registration method, characterized in that, Includes the following steps: Multimodal image data is acquired and imported into medical image processing software. The multimodal image data includes brain CT image data and brain magnetic resonance MRI image data stored in DICOM format. The medical image processing software includes 3DSlicer software. After the multimodal image data is imported into the 3DSlicer software, the rendering extension module in the 3DSlicer software is used to perform 3D rendering on the multimodal image data to obtain 3D rendered multimodal image data. Each of the data corresponding to one of the multimodal image data is designated as moving data, and the data corresponding to the other of the multimodal image data is designated as stationary data. The moving data is one of the 3D rendered multimodal image data, and the stationary data is the other of the 3D rendered multimodal image data. Planar correction is performed on the moving data and the stationary data respectively to obtain the registration plane of the corrected moving data and the registration plane of the corrected stationary data. The plane correction uses the crosshair positioning function in the 3DSlicer software. The registration plane includes the sagittal plane, the coronal plane, or the transverse plane. On the registration planes of the corrected moving data and the corrected stationary data, select any three points on non-corresponding straight lines. Establish a moving coordinate system and a stationary coordinate system based on the selected three points, and calculate the transformation matrix from the moving coordinate system to the stationary coordinate system. Specifically, this includes: The coordinates of three points on the registration plane of the corrected motion data and the registration plane of the corrected stationary data in the medical image processing software are obtained, namely t1, t2, and t3. The specific calculation formula is as follows: Calculate the unit vector along the z-axis: ； Calculate the unit vector along the x-axis: ； Calculate the unit vector along the y-axis: ； in, The norm of a vector or matrix; Based on the unit vectors of the x-axis, y-axis, and z-axis of the registration plane of the corrected moving data and the registration plane of the corrected stationary data, respectively establish the moving coordinate system Move and the stationary coordinate system Fix; The transformation matrix Trans is calculated using the following code: Move0=Move -1 ; Trans = Move0 ⋅ Fix; The transformation matrix is used to transform the moving data to obtain the transformed moving data. The transformed moving data and the stationary data are used to achieve pre-registration of the registration plane. The transformed motion data is fine-tuned to obtain the registration result of the multimodal image data. The fine-tuning includes single-axis rotation, bidirectional vertical translation, and bidirectional forward and backward translation. The axis of the single-axis rotation is perpendicular to the registration plane. The directions of the bidirectional vertical translation and the bidirectional forward and backward translation are perpendicular to each other and located on the registration plane.
2. The universal multimodal neural image registration method according to claim 1, characterized in that, Use the mark extension module in the 3DSlicer software to select any three points.
3. A universal multimodal neural image registration device, characterized in that, The general multimodal neural image registration method according to claim 1 or 2 includes: The import module is configured to acquire multimodal image data and import it into medical image processing software. The multimodal image data includes brain CT image data and brain magnetic resonance MRI image data stored in DICOM format. The correction module is configured to set data corresponding to one of the multimodal image data as moving data and data corresponding to the other of the multimodal image data as stationary data, and to perform planar correction on the moving data and the stationary data respectively to obtain the registration plane of the corrected moving data and the registration plane of the corrected stationary data. The pre-registration module is configured to select any three points on non-straight lines on the registration plane of the corrected moving data and the registration plane of the corrected stationary data, respectively, establish a moving coordinate system and a stationary coordinate system based on the selected three points, calculate the transformation matrix from the moving coordinate system to the stationary coordinate system, transform the moving data using the transformation matrix to obtain the transformed moving data, and achieve pre-registration of the registration plane using the transformed moving data and the stationary data. The fine-tuning module is configured to fine-tune the transformed motion data to obtain the registration result of the multimodal image data.
4. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 or 2.
5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in either claim 1 or 2.