Medical image registration method, system and device for corresponding relation missing problem

Abstract

The invention discloses a medical image registration method, a medical image registration system and a medical image registration device aiming at the problem of missing a corresponding relation, and relates to the technical field of image registration. Down-sampling the medical image data to be registered into a plurality of preoperative image data with different scale resolutions and follow-up image data with the initial scale resolution; constructing a registration network model containing a plurality of scale levels; performing feature extraction on the plurality of different scale resolutions to generate a deformation field; the attention correction module takes a deformation field generated in a previous level and preoperative lesion area weak supervision information as input, extracts attention from the current level after interpolating to the resolution of the current level, and performs weighted summation on an attention graph and a preoperative lesion area feature graph consistent with the resolution of the level in the feature extraction process of the level; generating a registered deformed image; according to the method, the problem that the final registration effect is affected due to step-by-step amplification of the initial registration error in multi-scale registration is relieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration, and in particular to a medical image registration method, system and device for the problem of missing correspondence. Background Art

[0002] Medical image registration is an important research direction in the field of medical image research and is widely used in preoperative surgical planning, surgical navigation, treatment response evaluation, and postoperative radiotherapy planning. Its goal is to establish the spatial transformation relationship between two or more images at different time points, different modalities, or different patients, so that medical images can be further studied and analyzed. Accurate registration of preoperative and postoperative follow-up images is crucial in the medical management of diseases such as gliomas, where surgical resection is the main treatment method, and is also conducive to promoting individualized treatment of patients. Taking glioma as an example, preoperative and follow-up images can be used as guiding information for evaluating the treatment effect of surgical resection and formulating further treatment strategies. However, due to the large deformation changes of brain tissue caused by various factors such as disease progression and surgery, the registration process is full of challenges.

[0003] Traditional registration methods are inadequate in handling the registration of paired MRI scans with high deformation variability and complexity, especially for the registration of preoperative and follow-up images of glioma patients, which have missing correspondences in both directions. In addition, the further development of the cavity created after surgical resection and the brain tissue around the resected area after surgery will lead to significant differences in brain anatomical structures, making the registration task more complicated. In recent years, learning-based multi-scale registration strategies have shown excellent results in various medical image registration tasks. However, few methods pay attention to the impact of initial scale registration errors on the overall registration and the existence of missing correspondences in multi-scale registration.

[0004] In summary, the existing multi-scale resolution registration methods have the problem that the registration error at the initial scale is gradually amplified, thus affecting the final registration effect. Summary of the invention

[0005] In view of the deficiency of the prior art that the registration error at the initial scale is gradually amplified, thus affecting the final registration effect, the present invention proposes a medical image registration method, system and device for the problem of missing correspondence, adopts a multi-scale resolution registration strategy, and realizes accurate deformation of large deformation areas by accurately generating a primary scale deformation field; introduces a correction attention module into the registration network, takes the preoperative lesion area as a form of weak supervision, and performs refined registration on possible missing corresponding areas, thereby solving the problems existing in the prior art.

[0006] A medical image registration method for the missing correspondence problem comprises the following steps:

[0007] Acquiring medical image data to be registered;

[0008] Downsampling the medical image data to be registered into a multi-scale resolution image pyramid; the multi-scale resolution image pyramid includes preoperative image data of multiple different scale resolutions and follow-up image data of the most initial scale resolution;

[0009] Extracting lesion area masks of a plurality of preoperative image data with different scale resolutions respectively;

[0010] The multi-scale resolution image pyramid is input into the registration network model of its corresponding multi-scale level. The preoperative image data corresponding to each scale level are respectively extracted through the stepped convolution layer to generate the corresponding deformation field. The deformation field generated in the previous level is used as the input of the correction attention module in the current level, and the deformation field is interpolated to the resolution of the current level to extract the attention map. The attention map extracted from each scale level and the mask feature map of the lesion area with the same resolution as the current level are weightedly summed in the feature extraction process of the current level until the final deformation field is generated in the topmost level. The final deformation field is applied to the follow-up image data through spatial transformation to generate the deformed image after registration.

[0011] Furthermore, the medical image data to be registered is sampled into a multi-scale resolution image pyramid through a trilinear interpolation down-sampling method.

[0012] Furthermore, the preoperative lesion region segmentation algorithm DMFNet is used to process the preoperative image data to obtain the lesion region mask.

[0013] Furthermore, the multi-scale hierarchical registration network model corresponding to the multi-scale resolution image pyramid is constructed through an end-to-end cascaded deep learning model.

[0014] Furthermore, the correction attention module includes three convolutional layers, two normalization layers, a nonlinear activation function and an activation function; the process of extracting the attention map through the correction attention module specifically includes the following steps:

[0015] The first convolutional layer is used to extract features from the interpolated deformation field;

[0016] The extracted features are passed through the normalization layer and then input into the second convolutional layer to further extract features; at the same time, nonlinear functions are introduced in the normalization process;

[0017] The lesion area mask is input into the third convolution layer for feature extraction, and the extracted feature map is passed through the normalization layer to obtain a lesion area mask feature map with the same resolution as the feature map extracted by the second convolution layer;

[0018] The mask feature map of the lesion area and the feature map extracted by the second convolutional layer are weighted fused to obtain the attention map.

[0019] The present invention also includes a medical image registration system for the missing correspondence problem, comprising:

[0020] An acquisition module, used for acquiring medical image data to be registered;

[0021] A downsampling module, used for downsampling the medical image data to be registered into a multi-scale resolution image pyramid; the multi-scale resolution image pyramid includes preoperative image data of multiple different scale resolutions and follow-up image data of the most initial scale resolution;

[0022] An extraction module, used for respectively extracting lesion area masks of a plurality of preoperative image data with different scale resolutions;

[0023] The registration module is used to input the multi-scale resolution image pyramid into the registration network model of its corresponding multi-scale level, and extract the features of the preoperative image data corresponding to each scale level through the stepped convolution layer to generate the corresponding deformation field; the deformation field generated in the previous level is used as the input of the correction attention module in the current level, and the deformation field is interpolated to the resolution of the current level, and then the attention map is extracted; the attention map extracted from each scale level and the mask feature map of the lesion area with the same resolution as the current level are weightedly summed in the feature extraction process of the current level until the final deformation field is generated in the topmost level; the final deformation field is applied to the follow-up image data through spatial transformation to generate a deformed image after registration.

[0024] The present invention also includes a medical image registration computer device for the problem of missing correspondence, including: a memory, a processor, and a computer program stored in the memory, and when the processor executes the computer program, the steps of the medical image registration method for the problem of missing correspondence are implemented.

[0025] The present invention also includes a readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, they are used to execute the steps of the medical image registration method for the missing correspondence problem.

[0026] The present invention provides a medical image registration method, system and device for the problem of missing correspondence, which has the following beneficial effects:

[0027] The present invention downsamples the medical image to be registered into preoperative image data of multiple scale resolutions and follow-up image data of the most initial scale resolution; constructs a registration network model containing multiple scale levels, and introduces a correction attention module, performs weighted summation on the attention map extracted at each scale level and the mask feature map of the lesion area with the same resolution as the current level in the feature extraction process of the current level, enhances the registration network's focus on areas where correspondence may be missing, generates a deformation field with accurate scale resolution, and avoids errors and offsets in the next level of registration; this method is superior to the existing strong baseline model in registration with missing correspondence, and alleviates the problem of the initial registration error in the multi-scale registration strategy being gradually amplified to affect the final registration effect from the perspective of quantitative evaluation indicators. This method has certain application significance for optimizing the registration of medical images with missing correspondence. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is an architecture diagram of the registration network model in an embodiment of the present invention;

[0029] Figure 2 This is a diagram of the architecture of the attention correction module in an embodiment of the present invention;

[0030] Figure 3 This is a comparison diagram of the effects of the present invention and the current mainstream registration method in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0032] The present invention proposes a medical image registration method for the problem of missing correspondence, which specifically comprises the following steps:

[0033] S1. Obtain medical images: obtain CT, MRI and other images through medical image acquisition equipment.

[0034] S2. Use a downsampling method to construct a multi-scale resolution image pyramid for the image to be registered, and obtain the image to be registered at multiple resolution scales; specifically, use a trilinear interpolation downsampling method to sample the image into an image pyramid of multiple resolution scales.

[0035] S3. Construct a multi-scale registration network model for feature extraction and deformation field generation, which includes stepped convolution. The multi-scale registration network is an end-to-end cascaded deep learning model that can predict and simultaneously generate deformation fields and deformed images in a single forward propagation. Stepped convolution is used to generate a relatively accurate deformation field in the initial level of registration and to maintain the accuracy of the deformation field at each scale in subsequent levels of registration.

[0036] S3.1. Design a correction attention module to extract attention from the deformation field of the previous level registration network. The specific architecture of the correction attention module is as follows: take the deformation field output from the previous level registration network of the model as input, first upsample it to the dimension of the current level registration network through trilinear interpolation, and then Figure 2 The convolution, normalization, and nonlinear activation operations shown on the right side of the figure extract attention about the degree of deformation.

[0037] S3.2. The attention extracted in the previous step is fused with the lesion area mask obtained by the preoperative lesion area segmentation algorithm DMFNet in the feature extraction process of the current level registration network to enhance the attention to large deformation areas and lesion areas; the extracted attention map and the preoperative lesion area information consistent with the current level dimension are weightedly summed in the feature extraction process of the current level registration.

[0038] S3.3. In the last level of registration, the deformation field is applied to the moving image through the STN spatial transformation network for spatial transformation to generate a deformed image; specifically, in the registration network with the maximum resolution scale, after the deformation field is generated, the deformation field is directly applied to the moving image through the spatial transformation network to generate a deformed image.

[0039] By combining steps S3 to S6 into an end-to-end deep learning network, prediction can be performed and deformation field and deformed image can be generated simultaneously in one forward propagation.

[0040] Example:

[0041] Reference Figure 1 As shown, a medical image registration method for the missing correspondence problem is used to register preoperative and follow-up images of glioma.

[0042] (1) Data acquisition: We analyzed the preoperative and follow-up images of 140 glioma patients from the Brain Tumor Sequence Registration (BraTS-Reg) Challenge and retrieved the magnetic resonance imaging data of these patients, which were required to include T1, T1-enhanced, T2, and T2FLAIR sequences.

[0043] (2) The fixed image and the moving image are sampled into an image pyramid containing three scale resolutions using the trilinear interpolation downsampling algorithm. According to the actual resolution and accuracy requirements, in this example, for one image, Among them I f1 ,I f2 ,I f3 Represents preoperative images with different scale resolutions, I m1 Follow-up images representing the highest initial scale resolution.

[0044] (3) Feature extraction and deformation field generation are performed through a multi-scale registration network. Figure 1 As shown, the network consists of a multi-scale registration network and a correction attention module.

[0045] The multi-scale registration network included in the present invention includes three scale levels. First, for The network uses larger convolution kernels to expand the receptive field, thereby obtaining relatively more accurate deformation features at the initial scale resolution and generating a deformation field.

[0046] (4) The attention of the next level registration network is extracted through the correction attention module. A relatively accurate initial scale deformation field is generated in the initial level registration network to avoid registration errors and offsets in the next level registration. To this end, the present invention proposes a correction attention module to complete the focus on the corresponding areas that may be missing. In this module, two steps are performed: attention extraction and preoperative lesion area information fusion.

[0047] For each level of the registration network, the deformation field of the previous level is After interpolating to the current resolution, the attention is extracted. After the attention is extracted, it is weighted and fused with the preoperative lesion area information of the same resolution.

[0048] (5) After the initial level of registration, the registration network combines the preoperative image at this level of resolution with the previous level deformed image W and deformation field sampled to the same resolution. The attention map generated by the correction attention module is taken as input, and the attention map is used to strengthen the attention on large deformation areas and lesion areas in the feature extraction process of the registration network at this level.

[0049] (6) Until the last level of the registration network, each level of the registration network can apply the attention map to the feature extraction process, thereby enhancing the network's attention to areas where correspondence may be missing, that is, areas with large deformation and lesion areas.

[0050] Therefore, the present invention can achieve accurate registration between medical images with missing correspondences. In order to compare with the mainstream registration methods, the present invention selects SyN, VoxelMorph and DIRAC as baseline methods, and the average values ​​of the quantitative indicators (target registration error) on the test set are 11.31, 8.43, 2.00 and 1.85 respectively; Figure 3 As shown, the present invention can generate more accurate registration results, thus facilitating the clinical application of deep registration models.

[0051] Based on the same inventive concept, the present invention also proposes a medical image registration system for the problem of missing correspondence, comprising:

[0052] The acquisition module is used to acquire the medical image data to be registered.

[0053] The downsampling module is used to downsample the medical image data to be registered into a multi-scale resolution image pyramid; the multi-scale resolution image pyramid includes preoperative image data of multiple different scale resolutions and follow-up image data of the most initial scale resolution.

[0054] The extraction module is used to respectively extract the lesion area masks of a plurality of preoperative image data with different scale resolutions.

[0055] The registration module is used to input the multi-scale resolution image pyramid into the registration network model of its corresponding multi-scale level, and extract the features of the preoperative image data corresponding to each scale level through the stepped convolution layer to generate the corresponding deformation field; the deformation field generated in the previous level is used as the input of the correction attention module in the current level, and the deformation field is interpolated to the resolution of the current level, and then the attention map is extracted; the attention map extracted from each scale level and the mask feature map of the lesion area with the same resolution as the current level are weightedly summed in the feature extraction process of the current level until the final deformation field is generated in the topmost level; the final deformation field is applied to the follow-up image data through spatial transformation to generate a deformed image after registration.

[0056] Based on the same inventive concept, the present invention also proposes a medical image registration computer device for the problem of missing correspondence, including: a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, it implements the steps of the medical image registration method for the problem of missing correspondence.

[0057] Based on the same inventive concept, the present invention also proposes a readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, they are used to execute the steps of a medical image registration method for the missing correspondence problem.

[0058] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A medical image registration method for the missing correspondence problem, characterized in that: The following steps are involved: Acquiring medical image data to be registered; Downsampling the medical image data to be registered into a multi-scale resolution image pyramid; The multi-scale resolution image pyramid includes preoperative image data of multiple different scale resolutions and follow-up image data of the most initial scale resolution; Extracting lesion area masks of a plurality of preoperative image data with different scale resolutions respectively; The multi-scale resolution image pyramid is input into the corresponding multi-scale level registration network model, and the preoperative image data corresponding to each scale level is respectively extracted through the stepped convolution layer to generate the corresponding deformation field; the deformation field generated in the previous level is used as the input of the correction attention module in the current level, and the deformation field is interpolated to the resolution of the current level, and then the attention map is extracted; the attention map extracted from each scale level and the lesion area mask feature map with the same resolution as the current level are weighted summed in the feature extraction process of the current level until the final deformation field is generated in the topmost level; The final deformation field is applied to the follow-up image data through spatial transformation to generate a registered deformed image.

2. A medical image registration method for the missing correspondence problem according to claim 1, characterized in that: The medical image data to be registered is sampled into a multi-scale resolution image pyramid through the trilinear interpolation down-sampling method.

3. A medical image registration method for the missing correspondence problem according to claim 1, characterized in that: The preoperative lesion region segmentation algorithm DMFNet is used to process the preoperative image data to obtain the lesion region mask.

4. The medical image registration method for the missing correspondence problem according to claim 1, characterized in that: The multi-scale hierarchical registration network model corresponding to the multi-scale resolution image pyramid is constructed through an end-to-end cascaded deep learning model.

5. The medical image registration method for the missing correspondence problem according to claim 3, characterized in that: The correction attention module includes three convolutional layers, two normalization layers, a nonlinear activation function and an activation function; the process of extracting the attention map through the correction attention module specifically includes the following steps: The first convolutional layer is used to extract features from the interpolated deformation field; The extracted features are passed through the normalization layer and then input into the second convolutional layer to further extract features; at the same time, nonlinear functions are introduced in the normalization process; The lesion area mask is input into the third convolution layer for feature extraction, and the extracted feature map is passed through the normalization layer to obtain a lesion area mask feature map with the same resolution as the feature map extracted by the second convolution layer; The mask feature map of the lesion area and the feature map extracted by the second convolutional layer are weighted fused to obtain the attention map.

6. A medical image registration system for the missing correspondence problem, characterized in that: include: An acquisition module, used for acquiring medical image data to be registered; A downsampling module, used for downsampling the medical image data to be registered into a multi-scale resolution image pyramid; The multi-scale resolution image pyramid includes preoperative image data of multiple different scale resolutions and follow-up image data of the most initial scale resolution; An extraction module, used for respectively extracting lesion area masks of a plurality of preoperative image data with different scale resolutions; A registration module is used to input the multi-scale resolution image pyramid into the registration network model of the corresponding multi-scale level, and extract the features of the preoperative image data corresponding to each scale level through the stepped convolution layer to generate the corresponding deformation field; the deformation field generated in the previous level is used as the input of the correction attention module in the current level, and the deformation field is interpolated to the resolution of the current level, and then the attention map is extracted; the attention map extracted from each scale level and the lesion area mask feature map with the same resolution as the current level are weighted summed in the feature extraction process of the current level until the final deformation field is generated in the topmost level; The final deformation field is applied to the follow-up image data through spatial transformation to generate a registered deformed image.

7. A medical image registration computer device for the missing correspondence problem, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein when the processor executes the computer program, the steps of the medical image registration method for the missing correspondence problem described in any one of claims 1 to 5 are implemented.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, they are used to execute the steps of the medical image registration method for the missing correspondence problem described in any one of claims 1 to 5.

Images