Medical image registration method and apparatus

Abstract

The application provides a medical image registration method and device. It is applied to the technical field of medical image processing. The method comprises the following steps: obtaining a to-be-registered image pair for medical image registration, the to-be-registered image pair comprising a floating image and a fixed image, the floating image being an image that needs to be subjected to spatial transformation in a registration operation, and the fixed image being a reference standard that is consistent with the spatial position and features of the floating image; performing hierarchical feature extraction on the floating image and the fixed image through a registration model to obtain a first multi-scale feature pyramid and a second multi-scale feature pyramid; performing correlation perception registration on the multi-scale features in the first multi-scale feature pyramid and the second multi-scale feature pyramid to obtain deformation field data; and performing spatial transformation on the floating image based on the deformation field data to obtain a target registration image. The application realizes efficient and accurate medical image registration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a medical image registration method and device. BACKGROUND

[0002] With the development of medical imaging equipment and clinical needs, medical image registration has become a key technology in auxiliary diagnosis and treatment, etc. The purpose is to align images of different sources to the same spatial coordinate system. In recent years, deep learning methods have made breakthroughs in deformable medical image registration. Registration methods based on convolutional neural networks can quickly predict high-quality deformation fields.

[0003] Although the prior art has made certain progress in the field of medical image registration, there are still obvious deficiencies. Some deep learning methods use global self-attention mechanisms to improve cross-region dependency modeling. However, these methods have high computational complexity and limited multi-scale information integration capability, which makes it difficult to meet the actual needs of registration efficiency and effectiveness in high-resolution three-dimensional medical image application scenarios. SUMMARY

[0004] Therefore, the present application provides a medical image registration method, which comprises: obtaining a pair of images to be registered for medical image registration, the pair of images to be registered comprising a floating image and a fixed image, the floating image being an image that needs to be spatially transformed in the registration operation, and the fixed image being a reference standard corresponding to the spatial position and features of the floating image in the registration operation; using a registration model to analyze the floating image and the fixed image to obtain deformation field data, wherein the registration model is used to extract hierarchical features from the floating image and the fixed image to obtain a first multi-scale feature pyramid and a second multi-scale feature pyramid, and the multi-scale features in the first multi-scale feature pyramid and the second multi-scale feature pyramid are subjected to correlation-aware registration to obtain the deformation field data; and performing spatial transformation on the floating image based on the deformation field data to obtain a target registration image.

[0005] Optionally, the registration model comprises a plurality of convolution modules, and the hierarchical feature extraction from the floating image and the fixed image by the registration model to obtain the first multi-scale feature pyramid and the second multi-scale feature pyramid comprises: using the plurality of convolution modules to perform continuous feature extraction on the floating image to obtain the first multi-scale feature pyramid, the first multi-scale feature pyramid comprising scale feature maps of the floating image extracted by each convolution module; and using the plurality of convolution modules to perform continuous feature extraction on the fixed image to obtain the second multi-scale feature pyramid, the second multi-scale feature pyramid comprising scale feature maps of the fixed image extracted by each convolution module.

[0006] Optionally, the registration model further comprises a first correlation-aware multi-window module layer, a second correlation-aware multi-window module layer, a deformable registration head layer, and a fusion layer; the correlation-aware registration of the multi-scale features in the first multi-scale feature pyramid and the second multi-scale feature pyramid is performed to obtain the deformation field data, comprising: the first correlation-aware multi-window module layer is used to respectively perform spatial correlation exploration on the corresponding scale feature maps in the first multi-scale feature pyramid and the second multi-scale feature pyramid to obtain scale feature correlation information at multiple levels; the second correlation-aware multi-window module layer is used to perform correlation exploration on the scale feature correlation information at each level and the scale feature correlation information at the previous level to obtain correlation features at multiple levels; the deformable registration head layer is used to respectively perform displacement field mapping on the correlation features at multiple levels to obtain residual displacement field data at multiple levels; and the fusion layer is used to continuously fuse the residual displacement field data at the corresponding level and the residual displacement field data at the previous level to obtain the deformation field data.

[0007] Optionally, the correlation-aware multi-window module in the first correlation-aware multi-window module layer comprises a three-dimensional correlation layer, a correlation-aware feature fusion layer, a multi-window module layer, a weighted fusion layer, and a residual channel attention module; the first correlation-aware multi-window module layer is used to respectively perform spatial correlation exploration on the corresponding scale feature maps in the first multi-scale feature pyramid and the second multi-scale feature pyramid to obtain scale feature correlation information at multiple levels, comprising: the three-dimensional correlation layer is used to perform block-by-block matching on local regions of the floating image scale feature map and the fixed image scale feature map, and calculate the correlation to obtain a three-dimensional correlation map; the correlation-aware feature fusion layer is used to splice and convolutionally fuse the floating image scale feature map, the fixed image scale feature map, and the three-dimensional correlation map to obtain a correlation-aware feature map; the multi-window module layer is used to extract multi-scale dependent information from the correlation-aware feature map to obtain a multi-scale correlation-aware feature map; the weighted fusion layer is used to perform channel weighted summation fusion on the multi-scale correlation-aware feature map to obtain a fused multi-scale correlation-aware feature; and the residual channel attention module is used to perform feature enhancement and channel attention adjustment on the fused multi-scale correlation-aware feature to obtain the scale feature correlation information at multiple levels.

[0008] Optionally, the floating image scale feature map and the fixed image scale feature map are matched in a local area by block in a three-dimensional correlation layer, and correlation is calculated to obtain a three-dimensional correlation map, including: based on a first preset stride, traversing sampling is performed on the floating image scale feature map to obtain a plurality of sampling points; based on preset three-dimensional block size information, a corresponding three-dimensional block is constructed with each sampling point as a center to obtain a plurality of first target three-dimensional blocks; based on a second preset stride, three-dimensional block traversing sampling is performed on the fixed image scale feature map in units of the first target three-dimensional blocks to obtain a plurality of second target three-dimensional blocks, the second preset stride being greater than the first preset stride; each first target three-dimensional block is matched with all second target three-dimensional blocks, and correlation of each first target three-dimensional block with all second target three-dimensional blocks is calculated to obtain the three-dimensional correlation map.

[0009] Optionally, the correlation of the first target three-dimensional block and the second target three-dimensional block is calculated in the following manner:

[0010] ,

[0011] wherein, represents the correlation, represents the size of the first target three-dimensional block and the second target three-dimensional block, represents the center voxel of the first target three-dimensional block, represents the center voxel of the second target three-dimensional block, represents will be iteratively traversed in the first target three-dimensional block where the represents will be iteratively traversed in the second target three-dimensional block where the represents represents the size of the first target three-dimensional block and the second target three-dimensional block.

[0012] Optionally, the size of the three-dimensional correlation map is:

[0013] ,

[0014] wherein, represents rounding down, represents the dimension of the floating image scale feature map and the fixed image scale feature map, represents the edge length of the first target three-dimensional block and the second target three-dimensional block, represents the first preset stride, represents the second preset stride, represents the number of channels of the three-dimensional correlation map.

[0015] Optionally, the multi-window module layer comprises a feature map region segmentation module, a visual state space module and a feature map region merging module; the multi-scale dependent information of the correlation-aware feature map is extracted by using the multi-window module layer to obtain a multi-scale correlation-aware feature map, comprising: the correlation-aware feature map is segmented into a plurality of non-overlapping region features by using the feature map region segmentation module to obtain a feature region map; the correlation of each region feature in the feature region map is captured by using the visual state space module to obtain a cross-region correlation feature map; and the cross-region correlation feature map is merged by using the feature map region merging module to obtain the multi-scale correlation-aware feature map.

[0016] Optionally, the medical image registration method provided by the application further comprises: pre-processing the floating image and the fixed image, comprising: truncating the floating image and the fixed image according to an intensity threshold value to obtain a truncated floating image and a truncated fixed image; performing intensity value normalization on the truncated floating image and the truncated fixed image according to the maximum intensity value and the minimum intensity value in the truncated floating image and the truncated fixed image to obtain a normalized floating image and a normalized fixed image; and resampling the normalized floating image and the normalized fixed image according to preset image size information to obtain a to-be-registered image pair.

[0017] The second aspect of the application provides a medical image registration device, comprising: a processor and a memory connected with the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to make the processor execute the medical image registration method described above.

[0018] The application obtains a to-be-registered image pair for medical image registration, which contains a floating image that needs to be spatially transformed and a fixed image as a reference standard; then a registration model is used to analyze the registration of the floating image and the fixed image, the registration model obtains a multi-scale feature pyramid through hierarchical feature extraction, and the image features are processed in different scales, which effectively reduces redundant calculation and reduces computational complexity. The multi-scale parallel processing adopts a coarse-to-fine registration strategy, the coarse scale can quickly locate the approximate matching area, and the fine scale can accurately adjust the registration, which significantly improves the registration efficiency. At the same time, the registration process fuses the correlation between images and between registration steps, the feature correlation between images provides guidance for accurately establishing spatial correspondence, and the correlation between steps provides important context information for each registration step, ensuring the accuracy of the registration process. After the above efficient and accurate registration analysis, the registration model outputs accurate deformation field data, and finally the floating image is spatially transformed based on the accurate deformation field data to obtain a target registration image that is highly consistent with the fixed image and accurate, which provides a solid and reliable image basis for subsequent medical diagnosis and treatment. BRIEF DESCRIPTION OF DRAWINGS

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the medical image registration method in an embodiment of the present invention;

[0021] Figure 2 This is a structural diagram of the registration model in an embodiment of the present invention;

[0022] Figure 3 This is a structural diagram of the related sensing multi-window module in an embodiment of the present invention. Detailed Implementation

[0023] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0025] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0026] like Figure 1 As shown, this embodiment of the invention provides a medical image registration method, which is executed by an electronic device such as a computer or server, and specifically includes:

[0027] S1, Obtain the image pair to be registered for medical image registration. The image pair to be registered includes a floating image and a fixed image. The floating image is the image that needs to be spatially transformed in the registration operation, and the fixed image is the reference standard that corresponds to the floating image in terms of spatial position and features in the registration operation.

[0028] The fixed image in the embodiment is a reference standard in the registration task, and the floating image is an object to be adjusted in the registration task, which may have differences in spatial position, direction, scale, etc. from the target state, and needs to be aligned with the fixed image through certain transformation operations (such as translation, rotation, scaling, etc.). What the floating image and the fixed image are needs to be determined according to the actual task, for example, it is required to perform registration of multi-modal MR (Magnetic Resonance) and CT (Computed Tomography), and it is required to register MR to CT, so the floating image is MR and the fixed image is CT.

[0029] S2, performing registration analysis on the floating image and the fixed image by using the registration model to obtain deformation field data, wherein the registration model is used to perform hierarchical feature extraction on the floating image and the fixed image to obtain a first multi-scale feature pyramid and a second multi-scale feature pyramid, and the multi-scale features in the first multi-scale feature pyramid and the second multi-scale feature pyramid are subjected to correlation perception registration to obtain the deformation field data.

[0030] Exemplarily, the training process of the registration model is as follows:

[0031] Obtaining training data pairs wherein the training data pairs include floating training images and fixed training images , n is the number of training data pairs, then the floating training images and the fixed training images are input into the pre-constructed registration network structure, and the registration network structure is trained in an end-to-end manner in an unsupervised manner, and the loss function used in the training can be expressed as: wherein, is used to measure the similarity between the deformed floating training image (the symbol can be understood as the process of applying the deformation field to the floating training image to obtain the registered image) and the fixed training image , is a regularization term used to constrain the deformation field , and is the weight of the regularization term, and the training is continued until the network converges to obtain the final registration model.

[0032] The floating image is subjected to hierarchical feature extraction by using a registration model to obtain a first multi-scale feature pyramid, and the fixed image is subjected to hierarchical feature extraction by using the registration model to obtain a second multi-scale feature pyramid. The registration model adopts a novel correlation-aware coarse-to-fine registration architecture, and multi-step registration is completed in one iteration. The architecture fuses the correlation between images and between registration steps as supplementary information. Based on the context information of the previous registration step, the multi-scale features in the first multi-scale feature pyramid and the second multi-scale feature pyramid are subjected to correlation-aware registration to obtain deformation field data capable of achieving optimal spatial matching between the floating image and the fixed image and reflecting spatial transformation information of each part of the floating image.

[0033] S3, performing spatial transformation on the floating image based on the deformation field data to obtain a target registration image.

[0034] According to the previously obtained deformation field data, the floating image is subjected to transformation operation in terms of position and shape in space to generate a target registration image that is registered with the fixed image.

[0035] In this embodiment, a pair of images to be registered for medical image registration is obtained, which includes a floating image that needs to be subjected to spatial transformation and a fixed image that serves as a reference standard. Then, the floating image and the fixed image are subjected to registration analysis by using a registration model. The registration model obtains a multi-scale feature pyramid through hierarchical feature extraction, and image features are processed in different scales in layers, which effectively reduces redundant calculation and reduces computational complexity. Multi-scale parallel processing adopts a coarse-to-fine registration strategy. Coarse scales can quickly locate the approximate matching region, and fine scales can accurately adjust the registration condition, which significantly improves the registration efficiency. At the same time, the registration process fuses the correlation between images and between registration steps. The feature correlation between images provides guidance for accurately establishing spatial correspondence, and the correlation between steps provides important context information for each registration step, which ensures the accuracy of the registration process. After the above efficient and accurate registration analysis, the registration model outputs accurate deformation field data. Finally, the floating image is subjected to spatial transformation based on the accurate deformation field data to obtain a target registration image that is highly consistent with the fixed image and is accurate, which provides a solid and reliable image basis for subsequent medical diagnosis and treatment.

[0036] In some optional embodiments of this embodiment, the registration model includes a plurality of convolution modules.

[0037] As Figure 2As shown, the registration model includes a hierarchical feature extraction encoder 1, which consists of four consecutive convolutional blocks. Each pair of convolutional blocks is connected by max pooling. Each convolutional block consists of two 3×3×3 convolutions, an activation function (LeakyReLU), and a normalization function (InstanceNorm).

[0038] Furthermore, step S2 involves extracting hierarchical features from the floating and fixed images using a registration model to obtain the first and second multi-scale feature pyramids. This process mainly includes:

[0039] S211, using multiple convolutional modules to continuously extract features from the floating image to obtain a first multi-scale feature pyramid, which includes the floating image scale feature maps extracted by each convolutional module.

[0040] like Figure 2 As shown, the first convolutional module in the hierarchical feature extraction encoder 1 processes the floating image. Perform convolution to output the first floating image scale feature map. The first floating image scale feature map is then processed through max pooling and reaches the second convolutional module. The second convolutional module then processes the max-pooled first floating image scale feature map. Perform convolution to output the second floating image scale feature map. The image is then processed through max pooling and reaches the third convolutional module. The third convolutional module then converts the max-pooled second floating image scale feature map... Perform convolution to output the third floating image scale feature map. The image is then processed through max pooling to reach the fourth convolutional module, which in turn processes the max-pooled third floating-scale feature map. Perform convolution to output the fourth floating image scale feature map. Finally, the first multi-scale feature pyramid is output by multiple convolutional modules in the encoder. ,in, It is the output of the i-th convolutional module.

[0041] S212, using multiple convolutional modules to extract continuous features from a fixed image, to obtain a second multi-scale feature pyramid, which includes fixed image scale feature maps extracted by each convolutional module.

[0042] like Figure 2 As shown, the first convolutional module in the hierarchical feature extraction encoder 1 processes a fixed image. Perform convolution to output the first fixed-scale image feature map. and is passed to a second convolution module through max-pooling, the second convolution module convolves the first fixed image scale feature map passed through max-pooling and outputs a second fixed image scale feature map and is passed to a third convolution module through max-pooling, the third convolution module convolves the second fixed image scale feature map passed through max-pooling and outputs a third fixed image scale feature map and is passed to a fourth convolution module through max-pooling, the fourth convolution module convolves the third fixed image scale feature map passed through max-pooling and outputs a fourth fixed image scale feature map a plurality of convolution modules in the final encoder outputs a second multi-scale feature pyramid wherein, is the output of the i-th convolution module.

[0043] The embodiment can obtain the first multi-scale feature pyramid and the second multi-scale feature pyramid containing scale feature maps extracted by the plurality of convolution modules in the encoder, respectively from the floating image and the fixed image, realize multi-scale expression of image features, and capture feature information of the image at different levels and scales. Meanwhile, max-pooling is used between adjacent convolution modules, which effectively reduces the dimension of the feature map, reduces the calculation amount, improves the calculation efficiency, provides rich and effective feature basis for subsequent registration analysis, and helps to improve the accuracy and efficiency of medical image registration.

[0044] In some optional embodiments of the embodiment, the registration model further includes a first correlation-aware multi-window module layer, a second correlation-aware multi-window module layer, a deformable registration head layer, and a fusion layer.

[0045] As shown in Figure 2 The registration model further includes a registration decoder 2 with correlation awareness and adopting a coarse-to-fine strategy, the registration decoder 2 includes a first correlation-aware multi-window module layer 21, a second correlation-aware multi-window module layer 22, a deformable registration head layer (Deformable Registration Head, DRH) 23, and a fusion layer 24. Figure 2and a fusion layer 23, wherein the first correlation-aware multi-window module layer 21 includes four correlation-aware multi-window (CMW-Mamba) modules for performing four-step coarse-to-fine registration analysis; the second correlation-aware multi-window module layer 22 includes three correlation-aware multi-window (CMW-Mamba) modules for performing three-step coarse-to-fine registration analysis; the deformable registration head layer includes four deformable registration heads for performing further processing and analysis on the feature output by the previous module to generate a corresponding residual displacement field; and the fusion layer 23 includes three fusion modules for adding the residual displacement field generated by the deformable registration head to the up-sampled residual displacement field, thereby updating the displacement field.

[0046] Further, the process of performing correlation-aware registration on the multi-scale features in the first multi-scale feature pyramid and the second multi-scale feature pyramid in step S2 to obtain the deformation field data mainly includes:

[0047] S221, using the first correlation-aware multi-window module layer to respectively perform spatial correlation exploration on the corresponding scale feature maps in the first multi-scale feature pyramid and the second multi-scale feature pyramid to obtain scale feature correlation information at multiple levels.

[0048] Using the extracted first multi-scale feature pyramid and second multi-scale feature pyramid, combined with the four correlation-aware multi-window (CMW-Mamba) modules in the first correlation-aware multi-window module layer 21, four-step coarse-to-fine registration analysis is performed to explore the spatial correlation of the scale feature maps.

[0049] S222, using the second correlation-aware multi-window module layer to perform correlation exploration on the scale feature correlation information at each level and the scale feature correlation information at the previous level to obtain correlation features at multiple levels.

[0050] Then, using the three correlation-aware multi-window (CMW-Mamba) modules in the second correlation-aware multi-window module layer 22, three-step coarse-to-fine registration analysis is performed to explore the correlation between steps.

[0051] S223, using the deformable registration head layer to respectively perform displacement field mapping on the correlation features at multiple levels to obtain residual displacement field data at multiple levels.

[0052] The deformable registration head (Registration Head) is composed of a convolution module, which uses the deformable registration head layer to perform convolution processing on the features output by the corresponding correlation-aware multi-window module in the second correlation-aware multi-window module layer 22 to generate a corresponding residual displacement field.

[0053] S224, using the fusion layer to continuously fuse the residual displacement field data of the corresponding level and the residual displacement field data of the previous level, to obtain the deformation field data.

[0054] Finally, the last fusion layer 23 adds the residual displacement field to the residual displacement field of the output of the previous level obtained by upsampling, and repeats the residual displacement field fusion step of each level to update the deformation field data.

[0055] Specifically, first, and are input into the corresponding fourth correlation-aware multi-window module in the first correlation-aware multi-window module layer 21, to explore the spatial correspondence between and The output features are then input into a deformable registration head, which maps the features to an initial displacement field The initial displacement field is used to guide the registration process in the second step, where the registered features ( represent the deformed feature map, such as W in Figure 2 ) are input into the corresponding third correlation-aware multi-window module in the first correlation-aware multi-window module layer 21, along with The output scale feature correlation information is input into the third correlation-aware multi-window module in the second correlation-aware multi-window module layer 22, after upsampling (such as the dashed arrow in Figure 2 , including transpose convolution and layer normalization), to explore the correlation between steps, and then the output correlation features are input into another deformable registration head to generate a residual displacement field. The residual displacement field is added to the upsampled initial displacement field to obtain an updated second displacement field . The above process is repeated twice, i.e., the third displacement field is obtained from and the fourth displacement field is obtained from and , i.e., the deformation field data.

[0056] ​The registration model in this embodiment comprises a first correlation-aware multi-window module layer, a second correlation-aware multi-window module layer, a deformable registration head layer, and a fusion layer working collaboratively to achieve correlation-aware registration of multi-scale features in the first and second multi-scale feature pyramids. Multiple correlation-aware multi-window modules in the first correlation-aware multi-window module layer perform coarse-to-fine registration analysis to explore spatial correlations in the scale feature maps. Multiple correlation-aware multi-window modules in the second correlation-aware multi-window module layer perform a three-step coarse-to-fine registration analysis to explore inter-step correlations. The deformable registration head layer performs displacement field mapping on the correlated features to obtain residual displacement field data, and the fusion layer continuously fuses the residual displacement field with the data from the previous level to obtain deformation field data. This multi-step registration analysis and feature fusion mechanism, from coarse to fine, can fully capture the spatial correspondence of image features and the correlation between steps, providing important contextual information for each registration step. This allows the model to perform more accurate registration in the current registration step based on its perception of the content completed in the previous step, and then gradually and accurately update the displacement field to finally obtain accurate deformation field data. This effectively achieves spatial alignment between floating and fixed images, improves the accuracy of deformation field data, and thus enhances the accuracy and reliability of medical image registration.

[0057] In some optional embodiments of this example, the correlation perception multi-window module in the first correlation perception multi-window module layer includes a three-dimensional correlation layer, a correlation perception feature fusion layer, a multi-window module layer, a weighted fusion layer, and a residual channel attention module.

[0058] like Figure 3 As shown, the first correlation-aware multi-window module layer 21 includes a 3D correlation layer, a correlation-aware feature fusion layer, a multi-window module layer, a weighted fusion layer (+, ×), and a residual channel attention module. The 3D correlation layer is used to construct an explicit voxel-level matching map, providing structural priors for subsequent feature alignment and deformation estimation. The correlation-aware feature fusion layer is used to concatenate and convolve the feature maps to further enhance the representation. The multi-window module layer is used to capture the correlation between feature points spanning a large area in the image. The residual channel attention module is used to further enhance key feature channels. Furthermore, the structure of the correlation-aware multi-window module in the second correlation-aware multi-window module layer 22 is the same as that in the first correlation-aware multi-window module layer 21, and will not be described again here.

[0059] Further, the process of using the first correlation-aware multi-window module layer to respectively explore the spatial correlation of the corresponding scale feature maps in the first multi-scale feature pyramid and the second multi-scale feature pyramid in step S221, to obtain scale feature correlation information at multiple levels, mainly includes:

[0060] S2211, using a three-dimensional correlation layer to perform block-by-block matching of local regions of the floating image scale feature map and the fixed image scale feature map, and calculating the correlation to obtain a three-dimensional correlation map.

[0061] The three-dimensional correlation layer of the present embodiment mainly performs block-by-block matching of local regions of the floating image feature and the fixed image feature within a 3D neighborhood. This operation can aggregate relevant features in the feature map, thereby emphasizing local details in the deep representation. and as an example, input and to the three-dimensional correlation layer for processing to obtain a three-dimensional correlation map. The dimensions of the input scale feature maps are all HxWxDxC in , where H represents the height of the input scale feature map, W represents the width of the input scale feature map, D represents the depth of the input scale feature map, and C in represents the number of channels of the input scale feature map.

[0062] S2212, using a correlation-aware feature fusion layer to splice and convolutionally fuse the floating image scale feature map, the fixed image scale feature map, and the three-dimensional correlation map to obtain a correlation-aware feature map.

[0063] Splice the three-dimensional correlation map and the input and , and fuse them through a 3x3x3 convolution layer to obtain a correlation-aware feature map . The dimensions of the output correlation-aware feature map are HxWxDxC out , where H represents the height of the output correlation-aware feature map , W represents the width of the output correlation-aware feature map , D represents the depth of the output correlation-aware feature map , and C out represents the number of channels of the output correlation-aware feature map .

[0064] S2213, using a multi-window module layer to extract multi-scale dependency information from the correlation-aware feature map to obtain a multi-scale correlation-aware feature map.

[0065] Splice the correlation-aware feature map The input is input into the multi-window module layer for multi-scale dependent information extraction to obtain a multi-scale correlation perception feature map.

[0066] In S2214, the multi-scale correlation perception feature map is fused by channel weighted summation using a weighted fusion layer to obtain a fused multi-scale correlation perception feature.

[0067] As shown in Figure 3 The multi-scale correlation perception feature map is input into the weighted fusion layer for channel weighted summation fusion, and the weight is dynamically learned by a global average pooling (GAP), two layers of dense connection and a softmax function.

[0068] In S2215, the fused multi-scale correlation perception feature is strengthened and the channel attention is adjusted using a residual channel attention module to obtain a plurality of hierarchical scale feature correlation information.

[0069] The residual channel attention module includes a layer normalization (Layer Norm), a convolution module (Conv), a leaky rectified linear unit (LeakyReLU) and an attention mechanism module (Squeeze-and-Excitation, SE).

[0070] In step S222, the second correlation perception multi-window module layer is used to explore the correlation of the scale feature correlation information of each hierarchical scale feature correlation information and the scale feature correlation information of the previous hierarchical scale feature correlation information, and the specific steps of obtaining the correlation feature of each hierarchical scale feature correlation information are the same as those in step S221, which will not be repeated here.

[0071] In the first correlation perception multi-window module layer or the second correlation perception multi-window module layer of the embodiment, the correlation perception multi-window module first obtains a three-dimensional correlation map from a three-dimensional correlation layer, then obtains a correlation perception feature map by convolution splicing of a correlation perception feature fusion layer, then a multi-window module layer extracts multi-scale dependent information, a weighted fusion layer performs channel weighted summation, and finally a residual channel attention module strengthens the feature and adjusts the channel attention, thereby realizing the spatial correlation exploration of the corresponding scale feature map in the first multi-scale feature pyramid and the second multi-scale feature pyramid. This process from constructing a matching map to gradually fusing and strengthening the feature can fully exploit the spatial correlation and multi-scale information of the image feature, effectively capture the local and global features of the image, provide rich, accurate and representative features for subsequent displacement field mapping and deformation field data generation, and thus improve the accuracy and reliability of medical image registration.

[0072] In some optional embodiments of this example, the process of using a three-dimensional correlation layer to perform block-by-block matching of local regions on the floating image scale feature map and the fixed image scale feature map, and calculating the correlation to obtain a three-dimensional correlation map, mainly includes:

[0073] S22111, based on the first preset stride, perform traversal sampling on the floating image scale feature map to obtain several sampling points.

[0074] First, use the first preset stride length. Sampling is performed on the floating image scale feature map to obtain several sampling points.

[0075] S22112, with each sampling point as the center, construct the corresponding three-dimensional block based on the preset three-dimensional block size information to obtain multiple first target three-dimensional blocks.

[0076] Then, based on the preset 3D block size information, the corresponding 3D block is constructed to obtain multiple blocks containing a central voxel. The first target three-dimensional block.

[0077] S22113, taking the first target 3D block as a unit, traversing and sampling the 3D block on the fixed image scale feature map based on the second preset stride to obtain multiple second target 3D blocks, the second preset stride being greater than the first preset stride.

[0078] Using the first target 3D block as the unit, and employing a second preset stride... Sampling is performed on a fixed-scale image feature map to obtain multiple samples containing a central voxel. The second target three-dimensional block.

[0079] S22114, match each first target 3D block with all second target 3D blocks respectively, and calculate the correlation between each first target 3D block and all second target 3D blocks respectively to obtain a 3D correlation map.

[0080] Specifically, first, the correlation between the first 3D block of the first target in the floating image scale feature map and all 3D blocks of the second target in the fixed image scale feature map is calculated. Then, the correlation between the second 3D block of the first target in the floating image scale feature map and all 3D blocks of the second target in the fixed image scale feature map is calculated, and so on, until the correlation between the last 3D block of the first target in the floating image scale feature map and all 3D blocks of the second target in the fixed image scale feature map is calculated. Based on all the correlations, a 3D correlation map is obtained. .

[0081] For example, the correlation between the first target 3D block and the second target 3D block is calculated using the following method:

[0082] ,

[0083] wherein, represents a correlation, represents a size of the first target three-dimensional block and the second target three-dimensional block, represents a center voxel of the first target three-dimensional block, represents a center voxel of the second target three-dimensional block, represents will be iteratively traversed within the first target three-dimensional block in which the is located, represents will be iteratively traversed within the second target three-dimensional block in which the is located.

[0084] wherein, the size of the three-dimensional correlation map is:

[0085] ,

[0086] wherein, represents a floor function, represents a dimension of the floating image scale feature map and the fixed image scale feature map, represents a side length of the first target three-dimensional block and the second target three-dimensional block, ), represents a first preset stride, represents a second preset stride, represents a number of channels of the three-dimensional correlation map.

[0087] The three-dimensional correlation layer of the embodiment can efficiently and accurately capture the local correspondence relationship between the floating image scale feature map and the fixed image scale feature map by sampling the target three-dimensional block in the floating image scale feature map and the fixed image scale feature map through different strides and matching and calculating the correlation. The three-dimensional correlation layer can capture the local pattern in the three-dimensional space, provide regional similarity constraints, generate a correlation map with the same shape as the floating image scale feature map and the fixed image scale feature map, ensure that the three can be stacked for processing, and provide a reliable foundation for the subsequent medical image registration process, thereby improving the registration accuracy.

[0088] In some optional embodiments of the embodiment, the multi-window module layer includes a feature map region segmentation module, a visual state space module, and a feature map region merging module.

[0089] The multi-window module layer includes N Mamba branches with M different window sizes, such as Figure 3As shown, the empirical setting N=3, i.e., three Mamba branches of different window sizes, in each Mamba branch, include a feature map region splitting module (Region Split), a visual state space module (VSS Block) and a feature map region merging module (Region Merge).

[0090] Further, the process of step S2213 of using the multi-window module layer to extract multi-scale dependent information from the correlation-aware feature map to obtain a multi-scale correlation-aware feature map mainly includes:

[0091] S22131, using a feature map region splitting module to split the correlation-aware feature map into a plurality of non-overlapping region features to obtain a feature region map.

[0092] As shown, Figure 3 the window sizes of the feature map region splitting modules in the three Mamba branches are 3x3x3, 5x5x5 and 7x7x7 respectively. Specifically, the correlation-aware feature map is input into the three feature map region splitting modules in parallel, and the feature map region splitting module divides the correlation-aware feature map into a plurality of non-overlapping regions according to the corresponding window size to obtain a feature region map of each branch.

[0093] S22132, using a visual state space module to capture the correlation of each region feature in the feature region map to obtain a cross-region correlation feature map.

[0094] The visual state space module can specifically include a layer normalization module (LN), a linear layer, a depthwise convolutional neural network module (DWCNN) and a selective scan module. The visual state space module has a linear complexity long-distance dependence modeling capability, which solves the computational bottleneck of traditional self-attention mechanisms in processing high-resolution visual data (especially 3D medical images). Moreover, combined with the multi-directional scanning of direction perception and the innovative multi-window division strategy, the visual state space module can more efficiently capture the correlation between feature points spanning a large range in medical images, solve the problem of large deformation and inter-organ correlation; it can also uniformly process large-scale deformation and local fine structure; at the same time, the visual state space module significantly reduces the computational and memory costs, and can process higher resolution data.

[0095] ​For each branch, the visual state space module is used to capture the correlation of each region feature in the input feature region map, obtaining a cross-region correlation feature map.

[0096] S22133, the feature region merging module is used to merge the cross-region correlation feature map, obtaining a multi-scale correlation perception feature map.

[0097] Finally, for the cross-region correlation feature map output by the visual state space module in each branch, the feature region merging module is used to merge the segmented regions, obtaining three multi-scale correlation perception feature maps.

[0098] The feature region segmentation module of the multiple branches in the embodiment can segment the correlation perception feature map into multiple non-overlapping region features according to different window sizes. Then, the visual state space module has long-distance dependence modeling capability with linear complexity, combined with multi-direction scanning and multi-window division strategy of direction perception, can efficiently capture the correlation between large-scale feature points in medical images, can uniformly process large-scale deformation and local fine structure, and significantly reduces the calculation and memory cost, and can process higher resolution data. Finally, the feature region merging module is used to merge the cross-region correlation feature map, obtaining a multi-scale correlation perception feature map, which improves the processing ability and efficiency of the model for multi-scale features of medical images.

[0099] In some optional embodiments of the embodiment, the medical image registration method further includes a pre-processing process for the floating image and the fixed image, and the pre-processing process specifically includes:

[0100] Step one, according to the intensity threshold, the floating image and the fixed image are truncated respectively, obtaining the truncated floating image and the truncated fixed image.

[0101] According to the pre-set intensity threshold, the floating image and the fixed image are processed, the pixel intensity values exceeding the threshold range in the image are adjusted to the intensity threshold boundary value, thereby obtaining the truncated floating image and the truncated fixed image. In this way, the abnormal high intensity or low intensity noise that may exist in the image can be removed, and the intensity distribution of the image is more concentrated in the meaningful range.

[0102] Step two, according to the intensity maximum value and the intensity minimum value in the truncated floating image and the truncated fixed image, the intensity value of the truncated floating image and the truncated fixed image is normalized respectively, obtaining the normalized floating image and the normalized fixed image.

[0103] After obtaining the truncated floating image and the truncated fixed image, the maximum and minimum intensity values of each are found, and then based on this, all pixel intensity values in the truncated floating image and the truncated fixed image are normalized to map the intensity values to a specific range (usually [0, 1]), to obtain the normalized floating image and the normalized fixed image. Normalization can eliminate the intensity differences between different images caused by factors such as acquisition equipment and environment.

[0104] Step three, resample the normalized floating image and the normalized fixed image according to the preset image size information to obtain the image pair to be registered.

[0105] Resample the normalized floating image and the normalized fixed image according to the preset image size information. Resampling refers to changing the pixel spacing and size of the image, so that the floating image and the fixed image have the same spatial resolution and size, and finally obtain the image pair to be registered. This step can ensure that the subsequent registration process is carried out at a unified spatial scale, improving the accuracy of registration.

[0106] The embodiment preprocesses the floating image and the fixed image to be registered, removes abnormal intensity noise in the image through intensity truncation, makes the image intensity distribution more reasonable, uses intensity value normalization to eliminate the intensity differences between different images, and provides an accurate basis for subsequent processing, and uses resampling to make the image have a unified spatial resolution and size, ensuring that the registration process is carried out at the same spatial scale. These preprocessing steps collectively improve the accuracy and efficiency of subsequent medical image registration, and provide a high-quality image pair to be registered for subsequent registration operations.

[0107] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0108] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the computer or other programmable data processing apparatus produce a means for implementing the functions described in the flowcharts and / or block diagrams. These computer program instructions can also be stored in a computer-usable storage medium that can guide a computer or other programmable data processing apparatus to work in a specific manner, so that the computer program instructions stored in the computer-usable storage medium produce a product for implementing the functions described in the flowcharts and / or block diagrams.Figure 1 one or more processes and / or blocks Figure 1 an apparatus with a means for performing the function specified by the block or blocks.

[0109] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the Figure 1 one or more processes and / or blocks Figure 1 an apparatus with a means for performing the function specified by the block or blocks.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 one or more processes and / or blocks Figure 1 an apparatus with a means for performing the function specified by the block or blocks.

[0111] Obviously, the above-described embodiments are only examples and are not intended to limit the present application. Other variations and modifications of the present application can be made based on the description disclosed herein. The present application is not to be limited to the embodiments disclosed herein but is only limited by the scope of the claims.

Claims

1. A medical image registration method, characterized in that, include: Acquire a pair of images to be registered for medical image registration. The pair of images to be registered includes a floating image and a fixed image. The floating image is the image that needs to be spatially transformed during the registration operation. The fixed image is a reference standard that corresponds to the floating image in terms of spatial position and features during the registration operation. The registration model is used to perform registration analysis on the floating image and the fixed image to obtain deformation field data. Specifically, the registration model is used to extract hierarchical features from the floating image and the fixed image to obtain a first multi-scale feature pyramid and a second multi-scale feature pyramid. Correlation-aware registration is performed on the multi-scale features in the first multi-scale feature pyramid and the second multi-scale feature pyramid to obtain the deformation field data. Based on the deformation field data, the floating image is spatially transformed to obtain the target registration image; The registration model includes multiple convolutional modules. The step of performing hierarchical feature extraction on the floating image and the fixed image using the registration model to obtain a first multi-scale feature pyramid and a second multi-scale feature pyramid includes: The floating image is continuously feature extracted using the multiple convolutional modules to obtain the first multi-scale feature pyramid, which includes the floating image scale feature maps extracted by each convolutional module. The fixed image is continuously feature extracted using the multiple convolutional modules to obtain the second multi-scale feature pyramid, which includes the fixed image scale feature maps extracted by each convolutional module. The registration model also includes a first correlation-sensing multi-window module layer, a second correlation-sensing multi-window module layer, a deformable registration head layer, and a fusion layer. The process of performing correlation-aware registration on the multi-scale features in the first and second multi-scale feature pyramids to obtain the deformation field data includes: The first related perception multi-window module layer is used to explore the spatial correlation of the corresponding scale feature maps in the first multi-scale feature pyramid and the second multi-scale feature pyramid respectively, so as to obtain scale feature related information at multiple levels. The second related perception multi-window module layer is used to explore the correlation between the scale feature related information of each level and the scale feature related information of the previous level, so as to obtain the correlation features of multiple levels. The deformable registration head layer is used to perform displacement field mapping on the correlation features of the multiple levels to obtain residual displacement field data of multiple levels. The deformation field data is obtained by continuously fusing the residual displacement field data of the corresponding level with the residual displacement field data of the previous level using the fusion layer.

2. The method according to claim 1, characterized in that, The first correlation perception multi-window module layer includes a three-dimensional correlation layer, a correlation perception feature fusion layer, a multi-window module layer, a weighted fusion layer, and a residual channel attention module; The first correlation perception multi-window module layer is used to explore the spatial correlation of the corresponding scale feature maps in the first multi-scale feature pyramid and the second multi-scale feature pyramid, respectively, to obtain scale feature-related information at multiple levels, including: The three-dimensional correlation layer is used to perform block-by-block matching of local regions on the floating image scale feature map and the fixed image scale feature map, and the correlation is calculated to obtain a three-dimensional correlation map; The correlation-aware feature fusion layer is used to stitch and convolve the floating image scale feature map, the fixed image scale feature map, and the three-dimensional correlation map to obtain the correlation-aware feature map. The multi-window module layer is used to extract multi-scale dependency information from the correlation-aware feature map to obtain a multi-scale correlation-aware feature map. The weighted fusion layer is used to perform channel-weighted summation and fusion on the multi-scale correlation sensing feature map to obtain the fused multi-scale correlation sensing features. The residual channel attention module is used to enhance the features and adjust the channel attention of the fused multi-scale correlation perception features to obtain the scale feature related information of the multiple levels.

3. The method according to claim 2, characterized in that, The step of using the three-dimensional correlation layer to perform block-by-block matching of local regions on the floating image scale feature map and the fixed image scale feature map, and calculating the correlation to obtain a three-dimensional correlation map, includes: Based on the first preset step size, traverse and sample on the floating image scale feature map to obtain a number of sampling points; With each sampling point as the center, a corresponding three-dimensional block is constructed based on the preset three-dimensional block size information to obtain multiple first target three-dimensional blocks; Using the first target 3D block as a unit, 3D block traversal sampling is performed on the fixed image scale feature map based on the second preset step size to obtain multiple second target 3D blocks, wherein the second preset step size is greater than the first preset step size; Each first target 3D block is matched with all second target 3D blocks, and the correlation between each first target 3D block and all second target 3D blocks is calculated to obtain the 3D correlation map.

4. The method according to claim 3, characterized in that, The correlation between the first target 3D block and the second target 3D block is calculated using the following method: , in, Indicates correlation. This indicates the size of the first target 3D block and the second target 3D block. This represents the central voxel of the first target 3D block. This represents the central voxel of the second target three-dimensional block. express Will Iterate through the first target 3D block where it is located. express Will Iterative traversal is performed within the second target 3D block.

5. The method according to claim 4, characterized in that, The dimensions of the three-dimensional correlation map are: ， in, This indicates rounding down. The dimensions represent the floating image scale feature map and the fixed image scale feature map. This represents the side lengths of the first target 3D block and the second target 3D block. This indicates the first preset step size. This indicates the second preset step size. This represents the number of channels in the three-dimensional correlation map.

6. The method according to claim 2, characterized in that, The multi-window module layer includes a feature map region segmentation module, a visual state space module, and a feature map region merging module. The step of extracting multi-scale dependency information from the correlation-aware feature map using the multi-window module layer to obtain a multi-scale correlation-aware feature map includes: The feature map region segmentation module is used to segment the correlation-aware feature map into multiple non-overlapping region features to obtain a feature region map. The visual state space module is used to capture the correlation of each region feature in the feature region map to obtain a cross-region correlation feature map. The cross-regional correlation feature maps are merged using the feature map region merging module to obtain the multi-scale correlation perception feature map.

7. The method according to claim 1, characterized in that, Also includes: Preprocessing of the floating image and the fixed image includes: The floating image and the fixed image are truncated according to the intensity threshold to obtain the truncated floating image and the truncated fixed image; The intensity values ​​of the truncated floating image and the truncated fixed image are normalized according to the maximum and minimum intensity values ​​in the truncated floating image and the truncated fixed image, respectively, to obtain normalized floating image and normalized fixed image; The normalized floating image and the normalized fixed image are resampled according to the preset image size information to obtain the image pair to be registered.

8. A medical image registration device, characterized in that, include: A processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to cause the processor to perform the medical image registration method as described in any one of claims 1-7.

Images