Multi-modal medical image registration method, device, equipment and medium

Abstract

The invention relates to a multi-modal medical image registration method and device, equipment and a medium, and the method comprises the steps: obtaining a first medical image and a second medical image; performing three-dimensional reconstruction on the first medical image based on the feature points of the first medical image to obtain a first three-dimensional structure; performing three-dimensional reconstruction on the second medical image based on the feature points of the second medical image to obtain a second three-dimensional structure; and registering the feature points of the first three-dimensional structure and the second three-dimensional structure to obtain a registered image. According to the invention, the registration precision of the medical image can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image registration, and in particular to a multimodal medical image registration method, device, equipment and medium. Background Art

[0002] Due to the limitation of imaging principle, existing medical imaging technology is only sensitive to tissues with certain characteristics. For example, computed tomography (CT) images can clearly show rigid features such as bones, while magnetic resonance imaging (MR) images have better representation of soft tissues. In medical image analysis, it is usually necessary to integrate these multimodal image information together for disease analysis and diagnosis. In this process, medical image registration technology plays a vital role.

[0003] Existing registration methods usually use heterogeneous registration fusion methods, but this method is only applicable to synchronous sampling. Since medical images of different modalities have different imaging principles, resolutions, and scanning positions, most of the acquisition of medical images of different modalities is performed separately, so it is not suitable for heterogeneous registration fusion methods. In addition, the existing heterogeneous registration fusion methods are difficult to produce satisfactory registration results for images with poor image quality, large position differences, and few image features. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a multimodal medical image registration method, device, equipment and medium, which can improve the registration accuracy of medical images.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a multimodal medical image registration method, comprising the following steps:

[0006] Acquire a first medical image and a second medical image; the first medical image and the second medical image are images of the same part of the same patient in different modalities;

[0007] Performing three-dimensional reconstruction on the first medical image based on feature points of the first medical image to obtain a first three-dimensional structure;

[0008] Performing three-dimensional reconstruction on the second medical image based on feature points of the second medical image to obtain a second three-dimensional structure;

[0009] The feature points of the first three-dimensional structure and the second three-dimensional structure are registered to obtain a registered image.

[0010] The three-dimensional reconstruction of the first medical image based on the feature points of the first medical image to obtain a first three-dimensional structure specifically includes:

[0011] Extracting rigid feature points and non-rigid feature points of the first medical image;

[0012] Filling the rigid region and the non-rigid region of the first medical image based on the rigid feature points and the non-rigid feature points of the first medical image;

[0013] The first medical image is three-dimensionally reconstructed using the filled rigid region and the non-rigid region to obtain a first three-dimensional structure.

[0014] Filling the rigid region and the non-rigid region of the first medical image based on the rigid feature points and the non-rigid feature points of the first medical image specifically includes:

[0015] Using the rigid feature points of the first medical image as first seed points;

[0016] Determine whether the similarity between the neighboring pixels of the pixel where the first seed point is located and the pixel where the first seed point is located exceeds a first threshold;

[0017] If the similarity between the neighboring pixels of the pixel where the first seed point is located and the pixel where the first seed point is located exceeds a first threshold, the neighboring pixels of the pixel where the first seed point is located are added to the rigid area, and the neighboring pixels of the pixel where the first seed point is located are used as new first seed point pixels, and the process returns to the previous step until the similarity between the neighboring pixels of the pixel where the first seed point is located and the pixel where the first seed point is located does not exceed the first threshold;

[0018] Using the non-rigid feature points of the first medical image as second seed points;

[0019] Determine whether the similarity between the neighboring pixels of the pixel where the second seed point is located and the pixel where the second seed point is located exceeds a second threshold;

[0020] If the similarity between the neighborhood pixels of the pixel where the second seed point is located and the pixel where the second seed point is located exceeds the second threshold, the neighborhood pixels of the pixel where the second seed point is located are added to the non-rigid area, and the neighborhood pixels of the pixel where the second seed point is located are used as new second seed point pixels, and the process returns to the previous step until the similarity between the neighborhood pixels of the pixel where the second seed point is located and the pixel where the second seed point is located does not exceed the second threshold.

[0021] The performing three-dimensional reconstruction on the second medical image based on the feature points of the second medical image to obtain a second three-dimensional structure specifically includes:

[0022] Extracting rigid feature points and non-rigid feature points of the second medical image;

[0023] Filling the rigid region and the non-rigid region of the second medical image based on the rigid feature points and the non-rigid feature points of the second medical image;

[0024] The filled rigid region and non-rigid region are used to perform three-dimensional reconstruction on the second medical image to obtain a second three-dimensional structure.

[0025] The filling of the rigid region and the non-rigid region of the second medical image based on the rigid feature points and the non-rigid feature points of the second medical image specifically includes:

[0026] Using the rigid feature points of the second medical image as third seed points;

[0027] Determine whether the similarity between the neighboring pixels of the pixel where the third seed point is located and the pixel where the third seed point is located exceeds a first threshold;

[0028] If the similarity between the neighboring pixels of the pixel where the third seed point is located and the pixel where the third seed point is located exceeds the first threshold, the neighboring pixels of the pixel where the third seed point is located are added to the rigid area, and the neighboring pixels of the pixel where the third seed point is located are used as new third seed point pixels, and the process returns to the previous step until the similarity between the neighboring pixels of the pixel where the third seed point is located and the pixel where the third seed point is located does not exceed the first threshold;

[0029] using the non-rigid feature points of the second medical image as fourth seed points;

[0030] Determine whether the similarity between the neighboring pixels of the pixel where the fourth seed point is located and the pixel where the fourth seed point is located exceeds a second threshold;

[0031] If the similarity between the neighboring pixels of the pixel where the fourth seed point is located and the pixel where the fourth seed point is located exceeds the second threshold, the neighboring pixels of the pixel where the fourth seed point is located are added to the non-rigid area, and the neighboring pixels of the pixel where the fourth seed point is located are used as new fourth seed point pixels, and the process returns to the previous step until the similarity between the neighboring pixels of the pixel where the fourth seed point is located and the pixel where the fourth seed point is located does not exceed the second threshold.

[0032] The registering the feature points of the first three-dimensional structure and the second three-dimensional structure to obtain a registered image specifically includes:

[0033] performing rigid registration on the rigid region of the first three-dimensional structure and the rigid region of the second three-dimensional structure;

[0034] Based on the result of the rigid registration, non-rigid registration is performed on the non-rigid features in the first three-dimensional structure and the second three-dimensional structure to obtain a registered image.

[0035] The rigid registration of the rigid region of the first three-dimensional structure and the rigid region of the second three-dimensional structure specifically includes:

[0036] Selecting new rigid feature points at the same position from the first three-dimensional structure and the second three-dimensional structure; wherein the new rigid feature points are feature points in the area where the image transformation gradient exceeds the gradient threshold;

[0037] constructing a transformation model based on coordinates of the new rigid feature points of the first three-dimensional structure and coordinates of the new rigid feature points of the second three-dimensional structure;

[0038] The transformation model is used to perform rigid registration on the rigid region of the first three-dimensional structure and the rigid region of the second three-dimensional structure.

[0039] The transformation model M is expressed as: Among them, a 11 、a 12 、a 13 、a 21 、a 22 、a 23 、a 31 、a 32 and a 33 is the rotation and scaling transformation parameter, a 14 、a 24 and a 34 is the translation transformation parameter.

[0040] When performing non-rigid registration on the non-rigid features in the first three-dimensional structure and the second three-dimensional structure, a deformation model based on B-spline is used for non-rigid registration.

[0041] The multimodal medical image registration method further includes: calculating the mutual information between the registered image and the first medical image or the second medical image, and evaluating the registration effect of the registered image according to the mutual information.

[0042] The technical solution adopted by the present invention to solve the technical problem is: to provide a multimodal medical image registration device, comprising:

[0043] An acquisition module, used to acquire a first medical image and a second medical image; the first medical image and the second medical image are images of the same part of the same patient in different modalities;

[0044] A first three-dimensional reconstruction module, connected to the acquisition module, for performing three-dimensional reconstruction on the first medical image based on feature points of the first medical image to obtain a first three-dimensional structure;

[0045] A second three-dimensional reconstruction module, connected to the acquisition module, for performing three-dimensional reconstruction on the second medical image based on feature points of the second medical image to obtain a second three-dimensional structure;

[0046] The registration module is connected to the first three-dimensional reconstruction module and the second three-dimensional reconstruction module respectively, and is used to register the feature points of the first three-dimensional structure and the second three-dimensional structure to obtain a registered image.

[0047] The technical solution adopted by the present invention to solve its technical problem is: to provide an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned multimodal medical image registration method when executing the computer program.

[0048] The technical solution adopted by the present invention to solve its technical problem is: providing a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned multimodal medical image registration method are implemented.

[0049] Beneficial Effects

[0050] Due to the adoption of the above-mentioned technical scheme, the present invention has the following advantages and positive effects compared with the prior art: the present invention reconstructs medical images of different modalities in three dimensions respectively by using rigid features and non-rigid features marked in medical images of different modalities, and then rigidly aligns the reconstructed three-dimensional models in combination with the rigid features, and then aligns the non-rigid features on the basis of the rigid registration, thereby improving the registration accuracy of the medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of a multimodal medical image registration method according to a first embodiment of the present invention;

[0052] Figure 2 is a registration effect diagram of the multimodal medical image registration method according to the first embodiment of the present invention;

[0053] Figure 3 Schematic diagram of a multimodal medical image registration device according to a second embodiment of the present invention.

[0054] Figure numerals: 1. acquisition module, 2. first three-dimensional reconstruction module, 21. first extraction unit, 22. first filling unit, 221. first seed determination subunit, 222. first judgment subunit, 223. first processing subunit, 224. second seed determination subunit, 225. second judgment subunit, 226. second processing subunit, 23. first reconstruction unit, 3. second three-dimensional reconstruction module, 31. second extraction unit, 32. second filling unit, 321. third seed determination subunit, 322. third judgment subunit, 323. third processing subunit, 324. fourth seed determination subunit, 325. fourth judgment subunit, 326. fourth processing subunit, 33. second reconstruction unit, 4. registration module, 41. rigid registration unit, 411. screening subunit, 412. construction subunit, 413. registration subunit, 42. non-rigid registration unit, 5. evaluation module. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0056] The first embodiment of the present invention relates to a multimodal medical image registration method, such as Figure 1 As shown, the following steps are included:

[0057] S1, obtaining a first medical image and a second medical image; the first medical image and the second medical image are images of the same part of the same patient in different modalities, for example, in this embodiment, the first medical image is a CT image of the patient's brain, and the second medical image is an MR image of the patient's brain. In order to ensure the accuracy of subsequent registration, a filtering algorithm such as mean filtering, median filtering, or Gaussian filtering can also be used to filter the medical image, thereby removing noise in the medical image.

[0058] S2, performing three-dimensional reconstruction on the first medical image based on the feature points of the first medical image to obtain a first three-dimensional structure, specifically comprising:

[0059] Extract rigid feature points and non-rigid feature points of the first medical image; wherein, rigid feature points refer to features that will not deform, such as a skull, and non-rigid feature points refer to features that will deform, such as water, soft tissue, etc.

[0060] Filling the rigid region and the non-rigid region of the first medical image based on the rigid feature points and the non-rigid feature points of the first medical image is specifically as follows:

[0061] Using the rigid feature points of the first medical image as first seed points;

[0062] Determine whether the similarity between the neighboring pixels of the pixel where the first seed point is located and the pixel where the first seed point is located exceeds a first threshold;

[0063] If the similarity between the neighboring pixels of the pixel where the first seed point is located and the pixel where the first seed point is located exceeds a first threshold, the neighboring pixels of the pixel where the first seed point is located are added to the rigid area, and the neighboring pixels of the pixel where the first seed point is located are used as new first seed point pixels, and the process returns to the previous step until the similarity between the neighboring pixels of the pixel where the first seed point is located and the pixel where the first seed point is located does not exceed the first threshold;

[0064] Using the non-rigid feature points of the first medical image as second seed points;

[0065] Determine whether the similarity between the neighboring pixels of the pixel where the second seed point is located and the pixel where the second seed point is located exceeds a second threshold;

[0066] If the similarity between the neighborhood pixels of the pixel where the second seed point is located and the pixel where the second seed point is located exceeds the second threshold, the neighborhood pixels of the pixel where the second seed point is located are added to the non-rigid area, and the neighborhood pixels of the pixel where the second seed point is located are used as new second seed point pixels, and the process returns to the previous step until the similarity between the neighborhood pixels of the pixel where the second seed point is located and the pixel where the second seed point is located does not exceed the second threshold.

[0067] The filled rigid region and the non-rigid region are used to perform three-dimensional reconstruction on the first medical image to obtain a first three-dimensional structure. During the reconstruction, the three-dimensional structure of the first medical image can be reconstructed using VTK software.

[0068] In this step, the first threshold and the second threshold can be set to be different, for example, the first threshold is 75-80%, and the second threshold is 85-90%. By setting different thresholds, the requirements for merging rigid features and non-rigid features can be met, making the filling of rigid areas and non-rigid areas more accurate.

[0069] S3, performing three-dimensional reconstruction on the second medical image based on the feature points of the second medical image to obtain a second three-dimensional structure, specifically comprising:

[0070] Extracting rigid feature points and non-rigid feature points of the second medical image.

[0071] Filling the rigid region and the non-rigid region of the second medical image based on the rigid feature points and the non-rigid feature points of the second medical image specifically includes:

[0072] Using the rigid feature points of the second medical image as third seed points;

[0073] Determine whether the similarity between the neighboring pixels of the pixel where the third seed point is located and the pixel where the third seed point is located exceeds a first threshold;

[0074] If the similarity between the neighboring pixels of the pixel where the third seed point is located and the pixel where the third seed point is located exceeds the first threshold, the neighboring pixels of the pixel where the third seed point is located are added to the rigid area, and the neighboring pixels of the pixel where the third seed point is located are used as new third seed point pixels, and the process returns to the previous step until the similarity between the neighboring pixels of the pixel where the third seed point is located and the pixel where the third seed point is located does not exceed the first threshold;

[0075] using the non-rigid feature points of the second medical image as fourth seed points;

[0076] Determine whether the similarity between the neighboring pixels of the pixel where the fourth seed point is located and the pixel where the fourth seed point is located exceeds a second threshold;

[0077] If the similarity between the neighboring pixels of the pixel where the fourth seed point is located and the pixel where the fourth seed point is located exceeds the second threshold, the neighboring pixels of the pixel where the fourth seed point is located are added to the non-rigid area, and the neighboring pixels of the pixel where the fourth seed point is located are used as new fourth seed point pixels, and the process returns to the previous step until the similarity between the neighboring pixels of the pixel where the fourth seed point is located and the pixel where the fourth seed point is located does not exceed the second threshold.

[0078] The second medical image is three-dimensionally reconstructed using the filled rigid region and the non-rigid region to obtain a second three-dimensional structure. During the reconstruction, the three-dimensional structure of the second medical image can also be reconstructed using VTK software. In this embodiment, the size of the three-dimensional structure of the second medical image is the same as the size of the three-dimensional structure of the first medical image.

[0079] Through the above method, the rigid feature areas and non-rigid feature areas of medical images can be automatically filled based on fewer feature points, providing a basis for constructing three-dimensional structures.

[0080] S4, registering the feature points of the first three-dimensional structure and the second three-dimensional structure to obtain a registered image (see Figure 2 ), including:

[0081] The rigid region of the first three-dimensional structure and the rigid region of the second three-dimensional structure are rigidly registered; when the rigid registration is performed, the following steps are specifically included:

[0082] New rigid feature points at the same position are selected from the first three-dimensional structure and the second three-dimensional structure; when selecting the new rigid feature points, feature points in the area with large image transformation gradient can be used as new rigid feature points.

[0083] A transformation model is constructed based on the coordinates of the new rigid feature points of the first three-dimensional structure and the coordinates of the new rigid feature points of the second three-dimensional structure. The obtained transformation model is expressed as: Among them, a 11 、a 12 、a 13 、a 21 、a 22 、a 23 、a 31 、a 32 and a 33 is the rotation and scaling transformation parameter, a 14 、a 24 and a 34 is the translation transformation parameter

[0084] The transformation model is used to perform rigid registration on the rigid area of ​​the first three-dimensional structure and the rigid area of ​​the second three-dimensional structure. The registration can be completed using y=Mx, where y is the coordinate of the rigid area of ​​the second three-dimensional structure and x is the coordinate of the rigid area of ​​the first three-dimensional structure.

[0085] In case of registration failure, the user can manually select new rigid feature points for reconfiguration or rescan the image for registration.

[0086] After completing the rigid registration, based on the result of the rigid registration, non-rigid features in the first three-dimensional structure and the second three-dimensional structure are non-rigidly registered to obtain a registered image. When performing non-rigid registration, this embodiment uses a deformation model based on B-spline for non-rigid registration.

[0087] It is not difficult to find that the present invention reconstructs medical images of different modalities in three dimensions by using rigid features and non-rigid features marked in medical images of different modalities, and then rigidly aligns the reconstructed three-dimensional models in combination with the rigid features, and then aligns the non-rigid features on the basis of the rigid registration, thereby improving the registration accuracy of medical images.

[0088] It is worth mentioning that, in order to evaluate the registration quality, the multimodal medical image registration method of this embodiment also includes: calculating the mutual information between the registered image and the first medical image or the second medical image, and evaluating the registration effect of the registered image according to the mutual information. The larger the calculated mutual information is, the better the registration effect is. Among them, the mutual information can measure the mutual dependence between the two images. The more similar the corresponding areas in the two images are, the higher their mutual information value is. Taking the registered image and the first medical image as an example, when calculating the mutual information between the two, the grayscale histogram of the registered image and the first medical image can be calculated first, and then the joint histogram of the registered image and the first medical image can be calculated, and then the mutual information between the two can be calculated using the obtained grayscale histogram and the joint histogram.

[0089] A second embodiment of the present invention relates to a multimodal medical image registration device, comprising:

[0090] An acquisition module 1 is used to acquire a first medical image and a second medical image; the first medical image and the second medical image are images of the same part of the same patient in different modalities;

[0091] A first three-dimensional reconstruction module 2, connected to the acquisition module 1, for performing three-dimensional reconstruction on the first medical image based on feature points of the first medical image to obtain a first three-dimensional structure;

[0092] A second three-dimensional reconstruction module 3, used to perform three-dimensional reconstruction on the second medical image based on feature points of the second medical image to obtain a second three-dimensional structure;

[0093] The registration module 4 is used to register the feature points of the first three-dimensional structure and the second three-dimensional structure to obtain a registered image.

[0094] The first 3D reconstruction module 2 comprises:

[0095] A first extraction unit 21, connected to the acquisition module 1, for extracting rigid feature points and non-rigid feature points of the first medical image;

[0096] A first filling unit 22, connected to the first extraction unit 21, is used to complete the filling of the rigid area and the non-rigid area of ​​the first medical image based on the rigid feature points and the non-rigid feature points of the first medical image;

[0097] The first reconstruction unit 23 is connected to the first filling unit 22, and is used to perform three-dimensional reconstruction on the first medical image by using the filled rigid area and non-rigid area to obtain a first three-dimensional structure.

[0098] The first filling unit 22 includes:

[0099] A first seed determination subunit 221, connected to the first extraction unit 21, configured to use the rigid feature points of the first medical image as first seed points;

[0100] A first judging subunit 222, connected to the first seed determining subunit 221, is used to judge whether the similarity between the neighboring pixels of the pixel where the first seed point is located and the pixel where the first seed point is located exceeds a first threshold;

[0101] The first processing subunit 223 is connected to the first judging subunit 222, and is used to add the neighboring pixels of the pixel where the first seed point is located to the rigid area, and use the neighboring pixels of the pixel where the first seed point is located as new first seed point pixels, and return to the first judging subunit 222, until the similarity between the neighboring pixels of the pixel where the first seed point is located and the pixel where the first seed point is located does not exceed the first threshold;

[0102] A second seed determination subunit 224, connected to the first extraction unit 21, for using the non-rigid feature points of the first medical image as second seed points;

[0103] The second judging subunit 225 is connected to the second seed determining subunit 224, and is used to judge whether the similarity between the neighboring pixels of the pixel where the second seed point is located and the pixel where the second seed point is located exceeds a second threshold;

[0104] The second processing sub-unit 226 is connected to the second judgment sub-unit 225, and is used to add the neighborhood pixels of the pixel where the second seed point is located to the non-rigid area, and use the neighborhood pixels of the pixel where the second seed point is located as new second seed point pixels, and return to the second judgment sub-unit 225 until the similarity between the neighborhood pixels of the pixel where the second seed point is located and the pixel where the second seed point is located does not exceed the second threshold.

[0105] The second 3D reconstruction module 3 comprises:

[0106] A second extraction unit 31, connected to the acquisition module 1, for extracting rigid feature points and non-rigid feature points of the second medical image;

[0107] A second filling unit 32, connected to the second extraction unit 31, is used to complete the filling of the rigid area and the non-rigid area of ​​the second medical image based on the rigid feature points and the non-rigid feature points of the second medical image;

[0108] The second reconstruction unit 33 is connected to the second filling unit 32, and is used to perform three-dimensional reconstruction on the second medical image by using the filled rigid area and non-rigid area to obtain a second three-dimensional structure.

[0109] The second filling unit 32 includes:

[0110] A third seed determination subunit 321, connected to the second extraction unit 31, for using the rigid feature points of the second medical image as third seed points;

[0111] The third judging subunit 322 is connected to the third seed determining subunit 321, and judges whether the similarity between the neighboring pixels of the pixel where the third seed point is located and the pixel where the third seed point is located exceeds a third threshold;

[0112] The third processing subunit 323 is connected to the third judging subunit 322, and is used to add the neighboring pixels of the pixel where the third seed point is located to the rigid area, and use the neighboring pixels of the pixel where the third seed point is located as new third seed point pixels, and return to the third judging subunit 322, when the similarity between the neighboring pixels of the pixel where the third seed point is located and the pixel where the third seed point is located exceeds the third threshold, until the similarity between the neighboring pixels of the pixel where the third seed point is located and the pixel where the third seed point is located does not exceed the third threshold;

[0113] a fourth seed determination subunit 324, connected to the second extraction unit 31, and configured to use the non-rigid feature points of the second medical image as fourth seed points;

[0114] The fourth judging subunit 325 is connected to the fourth seed determining subunit 324, and is used to judge whether the similarity between the neighboring pixels of the pixel where the fourth seed point is located and the pixel where the fourth seed point is located exceeds a threshold;

[0115] The fourth processing sub-unit 326 is connected to the fourth judgment sub-unit 325, and is used to add the neighboring pixels of the pixel where the fourth seed point is located to the non-rigid area, and use the neighboring pixels of the pixel where the fourth seed point is located as new fourth seed point pixels, and return to the fourth judgment sub-unit 325 until the similarity between the neighboring pixels of the pixel where the fourth seed point is located and the pixel where the fourth seed point is located does not exceed the threshold.

[0116] The registration module 4 comprises:

[0117] A rigid registration unit 41, connected to the first 3D reconstruction module 2 and the second 3D reconstruction module 3, respectively, for performing rigid registration on the rigid region of the first 3D structure and the rigid region of the second 3D structure;

[0118] The non-rigid registration unit 42 is used to perform non-rigid registration on the non-rigid features in the first three-dimensional structure and the second three-dimensional structure based on the result of the rigid registration to obtain a registered image.

[0119] The rigid registration unit 41 comprises:

[0120] A screening subunit 411 is used to select new rigid feature points at the same position from the first three-dimensional structure and the second three-dimensional structure; wherein the new rigid feature points are feature points in the area where the image transformation gradient exceeds the gradient threshold;

[0121] A construction subunit 412, configured to construct a transformation model based on the coordinates of the new rigid feature points of the first three-dimensional structure and the coordinates of the new rigid feature points of the second three-dimensional structure;

[0122] The registration subunit 413 is configured to perform rigid registration on the rigid region of the first three-dimensional structure and the rigid region of the second three-dimensional structure by using the transformation model.

[0123] The transformation model M is expressed as: Among them, a 11 、a 12 、a 13 、a 21 、a 22 、a 23 、a 31 、a 32 and a 33 is the rotation and scaling transformation parameter, a 14 、a 24 and a 34 is the translation transformation parameter.

[0124] The non-rigid registration unit 42 performs non-rigid registration using a deformation model based on B-spline.

[0125] The multimodal medical image registration device further includes: an evaluation module 5, which is used to calculate the mutual information between the registered image and the first medical image or the second medical image, and evaluate the registration effect of the registered image according to the mutual information.

[0126] A third embodiment of the present invention relates to an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multimodal medical image registration method of the first embodiment when executing the computer program.

[0127] A fourth embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the multimodal medical image registration method of the first embodiment.

[0128] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0129] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction method, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0132] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal medical image registration method, characterized in that: The following steps are involved: Acquire a first medical image and a second medical image; the first medical image and the second medical image are images of the same part of the same patient in different modalities; Performing three-dimensional reconstruction on the first medical image based on feature points of the first medical image to obtain a first three-dimensional structure; Performing three-dimensional reconstruction on the second medical image based on feature points of the second medical image to obtain a second three-dimensional structure; The feature points of the first three-dimensional structure and the second three-dimensional structure are registered to obtain a registered image.

2. The multimodal medical image registration method according to claim 1, characterized in that: The three-dimensional reconstruction of the first medical image based on the feature points of the first medical image to obtain a first three-dimensional structure specifically includes: extracting rigid feature points and non-rigid feature points of the first medical image; Filling the rigid region and the non-rigid region of the first medical image based on the rigid feature points and the non-rigid feature points of the first medical image; The first medical image is three-dimensionally reconstructed using the filled rigid region and the non-rigid region to obtain a first three-dimensional structure.

3. The multimodal medical image registration method according to claim 2, characterized in that: Filling the rigid region and the non-rigid region of the first medical image based on the rigid feature points and the non-rigid feature points of the first medical image specifically includes: Using the rigid feature points of the first medical image as first seed points; Determine whether the similarity between the neighboring pixels of the pixel where the first seed point is located and the pixel where the first seed point is located exceeds a first threshold; If the similarity between the neighboring pixels of the pixel where the first seed point is located and the pixel where the first seed point is located exceeds a first threshold, the neighboring pixels of the pixel where the first seed point is located are added to the rigid area, and the neighboring pixels of the pixel where the first seed point is located are used as new first seed point pixels, and the process returns to the previous step until the similarity between the neighboring pixels of the pixel where the first seed point is located and the pixel where the first seed point is located does not exceed the first threshold; Using the non-rigid feature points of the first medical image as second seed points; Determine whether the similarity between the neighboring pixels of the pixel where the second seed point is located and the pixel where the second seed point is located exceeds a second threshold; If the similarity between the neighborhood pixels of the pixel where the second seed point is located and the pixel where the second seed point is located exceeds the second threshold, the neighborhood pixels of the pixel where the second seed point is located are added to the non-rigid area, and the neighborhood pixels of the pixel where the second seed point is located are used as new second seed point pixels, and the process returns to the previous step until the similarity between the neighborhood pixels of the pixel where the second seed point is located and the pixel where the second seed point is located does not exceed the second threshold.

4. The multimodal medical image registration method according to claim 1, characterized in that: The three-dimensional reconstruction of the second medical image based on the feature points of the second medical image to obtain the second three-dimensional structure specifically includes: extracting rigid feature points and non-rigid feature points of the second medical image; Filling the rigid region and the non-rigid region of the second medical image based on the rigid feature points and the non-rigid feature points of the second medical image; The filled rigid region and non-rigid region are used to perform three-dimensional reconstruction on the second medical image to obtain a second three-dimensional structure.

5. The multimodal medical image registration method according to claim 4, characterized in that: The filling of the rigid region and the non-rigid region of the second medical image based on the rigid feature points and the non-rigid feature points of the second medical image specifically includes: Using the rigid feature points of the second medical image as third seed points; Determine whether the similarity between the neighboring pixels of the pixel where the third seed point is located and the pixel where the third seed point is located exceeds a first threshold; If the similarity between the neighboring pixels of the pixel where the third seed point is located and the pixel where the third seed point is located exceeds the first threshold, the neighboring pixels of the pixel where the third seed point is located are added to the rigid area, and the neighboring pixels of the pixel where the third seed point is located are used as new third seed point pixels, and the process returns to the previous step until the similarity between the neighboring pixels of the pixel where the third seed point is located and the pixel where the third seed point is located does not exceed the first threshold; using the non-rigid feature points of the second medical image as fourth seed points; Determine whether the similarity between the neighboring pixels of the pixel where the fourth seed point is located and the pixel where the fourth seed point is located exceeds a second threshold; If the similarity between the neighboring pixels of the pixel where the fourth seed point is located and the pixel where the fourth seed point is located exceeds the second threshold, the neighboring pixels of the pixel where the fourth seed point is located are added to the non-rigid area, and the neighboring pixels of the pixel where the fourth seed point is located are used as new fourth seed point pixels, and the process returns to the previous step until the similarity between the neighboring pixels of the pixel where the fourth seed point is located and the pixel where the fourth seed point is located does not exceed the second threshold.

6. The multimodal medical image registration method according to claim 1, characterized in that: The registering the feature points of the first three-dimensional structure and the second three-dimensional structure to obtain a registered image specifically includes: performing rigid registration on the rigid region of the first three-dimensional structure and the rigid region of the second three-dimensional structure; Based on the result of the rigid registration, non-rigid registration is performed on the non-rigid features in the first three-dimensional structure and the second three-dimensional structure to obtain a registered image.

7. The multimodal medical image registration method according to claim 6, characterized in that: The rigid registration of the rigid region of the first three-dimensional structure and the rigid region of the second three-dimensional structure specifically includes: Selecting new rigid feature points at the same position from the first three-dimensional structure and the second three-dimensional structure; wherein the new rigid feature points are feature points in the area where the image transformation gradient exceeds the gradient threshold; constructing a transformation model based on coordinates of the new rigid feature points of the first three-dimensional structure and coordinates of the new rigid feature points of the second three-dimensional structure; The transformation model is used to perform rigid registration on the rigid region of the first three-dimensional structure and the rigid region of the second three-dimensional structure.

8. The multimodal medical image registration method according to claim 7, characterized in that: The transformation model M is expressed as: Among them, a 11 、a 12 、a 13 、a 21 、a 22 、a 23 、a 31 、a 32 and a 33 is the rotation and scaling transformation parameter, a 14 、a 24 and a 34 is the translation transformation parameter.

9. The multimodal medical image registration method according to claim 6, characterized in that: When performing non-rigid registration on the non-rigid features in the first three-dimensional structure and the second three-dimensional structure, a deformation model based on B-spline is used for non-rigid registration.

10. The multimodal medical image registration method according to claim 1, characterized in that: Also includes: The mutual information between the registered image and the first medical image or the second medical image is calculated, and the registration effect of the registered image is evaluated according to the mutual information.

11. A multimodal medical image registration device, characterized in that: include: An acquisition module, used to acquire a first medical image and a second medical image; the first medical image and the second medical image are images of the same part of the same patient in different modalities; A first three-dimensional reconstruction module, connected to the acquisition module, configured to perform three-dimensional reconstruction on the first medical image based on feature points of the first medical image to obtain a first three-dimensional structure; A second three-dimensional reconstruction module, connected to the acquisition module, for performing three-dimensional reconstruction on the second medical image based on feature points of the second medical image to obtain a second three-dimensional structure; The registration module is connected to the first three-dimensional reconstruction module and the second three-dimensional reconstruction module respectively, and is used to register the feature points of the first three-dimensional structure and the second three-dimensional structure to obtain a registered image.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multimodal medical image registration method as described in any one of claims 1-10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal medical image registration method as described in any one of claims 1-10 are implemented.

Images