A Large-Deformation Medical Image Registration Method Based on Multi-Window Gated Multilayer Perceptron and Context Residual Enhancement

By employing a multi-window gated multilayer perceptron and contextual residual enhancement, the high computational complexity and error accumulation issues in medical image registration with complex nonlinear large deformations are addressed. This approach enables high-precision, low-cost real-time medical image registration, suitable for complex deformation scenarios.

CN118982566BActive Publication Date: 2026-11-17ANHUI UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411064282.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2026-11-17
Estimated Expiration
2044-08-05

AI Technical Summary

Technical Problem

Existing medical image registration methods suffer from high computational complexity and insufficient accuracy when dealing with complex nonlinear large deformations. Multi-level pyramid methods accumulate errors, and the degrees of freedom of low-resolution deformation fields are limited, making it difficult to meet the requirements of real-time clinical applications and high precision.

Method used

A multi-window gated multilayer perceptron and context residual enhancement method are adopted. Feature maps are extracted by an improved dual-stream shared weight encoder, and residual deformation fields are generated by combining 3D correlation layers and multi-window gated multilayer perceptron. The feature maps are enhanced by the context residual enhancement module to generate a high-precision total residual deformation field, ensuring the rationality and consistency of the deformation process.

Benefits of technology

It improves the accuracy and computational efficiency of medical image registration, can handle complex nonlinear large deformations, avoids error accumulation, and is suitable for real-time applications. It performs particularly well in complex and multi-scale deformations, and improves the Dice coefficient to 0.870980.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982566B_ABST
    Figure CN118982566B_ABST
Patent Text Reader

Abstract

This invention relates to a method for large-deformation medical image registration based on a multi-window gated multilayer perceptron and contextual residual enhancement. The method includes: acquiring a three-dimensional medical image dataset and dividing it into floating and fixed images; obtaining four layers of floating feature maps and four layers of fixed feature maps with different resolutions; obtaining a fused feature map; outputting the enhanced feature map; obtaining a target total residual deformation field; and distorting the image to be registered using the target total residual deformation field to obtain the final registration result. This invention uses a multi-window gated multilayer perceptron to capture sub-deformation fields from different receptive fields, enabling the network to capture scale information and spatial details more precisely, resulting in higher computational efficiency and suitability for real-time applications. By scaling and squaring the velocity field, the smoothness and continuity of the deformation are ensured, avoiding the discontinuities and inconsistencies that may arise from simply adding the deformation field together. This method is suitable for handling complex large-deformation medical image registration tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer vision technology, and in particular to a large-deformation medical image registration method based on multi-window gated multilayer perceptron and contextual residual enhancement. Background Technology

[0002] Medical image registration maps voxels from different images to the same coordinate system, aligning anatomical structures in images from different states or individuals. It is widely used in various clinical scenarios, such as surgical planning, assisted diagnosis, and pathological analysis. In these clinical applications, different images typically exhibit complex deformation relationships, stemming from individual anatomical differences, pose changes during imaging, and variations in image acquisition equipment. Traditional medical image registration methods usually utilize iterative optimization of energy functions to gradually align anatomical structures between images. This iterative computation is computationally complex and unsuitable for real-time clinical applications. In recent years, deep learning technology has driven the development of medical image registration, with registration methods based on convolutional neural networks and Transformers emerging. However, most of these methods attempt to directly predict the final deformation field in a single registration step, which can lead to insufficient description of the deformation field of images with large nonlinear deformation relationships.

[0003] To address this issue, some registration methods employ a strategy of cascading multiple U-shaped networks to predict multiple sub-deformation fields with the same resolution, gradually processing complex deformation problems. However, this strategy increases the computational burden and is insufficient for handling multi-scale deformations. To address this, pyramid-based methods have been proposed. These methods estimate deformation fields with different resolutions by constructing a feature pyramid strategy. The low-resolution deformation field is used to handle global deformation, while the high-resolution deformation field focuses on handling detailed deformations. However, while existing pyramid-based registration methods utilize inter-layer multi-scale fields to coarsely decompose deformation, the problem of large intra-layer deformation components remains. Furthermore, at low resolutions, accurately representing such complex motion patterns is extremely challenging due to the limited degrees of freedom of a single deformation field. Additionally, some pyramid methods attempt to generate sub-deformation fields through multiple cascades within each layer. This method suffers from error accumulation, and its registration performance is heavily dependent on the number of iterations.

[0004] Therefore, improving the pyramid registration method to handle complex deformations and enhance the accuracy of unsupervised registration is the key to solving the problem of complex nonlinear large deformations in medical images. Summary of the Invention

[0005] To address the problems of high computational complexity, insufficient handling of complex nonlinear large deformations, error accumulation in multi-level pyramid methods, and limited degrees of freedom in low-resolution deformation fields in existing technologies, the present invention aims to provide a large deformation medical image registration method based on multi-window gated multilayer perceptron and context residual enhancement, which is highly accurate, computationally efficient, ensures the rationality and consistency of the deformation process, and is suitable for handling complex medical image registration tasks.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a large-deformation medical image registration method based on multi-window gated multilayer perceptron and context residual enhancement, the method comprising the following sequential steps:

[0007] (1) Obtain a three-dimensional medical image dataset, preprocess it, and divide the preprocessed three-dimensional medical image dataset into floating images and fixed images;

[0008] (2) Input the floating image and the fixed image into the improved dual-stream shared weight encoder for feature extraction to obtain four floating feature maps with different resolutions and four fixed feature maps with different resolutions.

[0009] (3) The four floating feature maps with different resolutions and the four fixed feature maps with different resolutions are used to generate local correlation feature maps through the 3D correlation layer. The local correlation feature maps, floating feature maps and fixed feature maps are then fused to obtain a fused feature map.

[0010] (4) After upsampling, the local related feature map is connected to the fused feature map at the same resolution along the channel dimension, and then input into the context residual enhancement module. The context residual enhancement module outputs the enhanced feature map.

[0011] (5) The feature maps of each layer are enhanced to generate residual deformation fields through a multi-window gated multilayer perceptron. The residual deformation fields are then upsampled and fused with the total residual deformation fields to obtain the target total residual deformation field.

[0012] (6) The image to be registered is distorted by the total residual deformation field of the target to obtain the final registration result.

[0013] In step (1), the preprocessing includes affine transformation, cropping, and brightness normalization.

[0014] Step (2) specifically includes the following steps:

[0015] (2a) Construct an improved dual-stream shared weight encoder, which consists of four consecutive convolutional blocks, namely the first convolutional block, the second convolutional block, the third convolutional block and the fourth convolutional block. Adjacent convolutional blocks are connected by an average pooling layer. The four convolutional blocks have the same structure. Each convolutional block contains two 3D convolutional layers with a kernel size of 3×3×3 and a stride of 1, as well as a LeakyReLU activation function and an instance normalization layer.

[0016] (2b) The floating image and the fixed image are respectively input into the improved dual-stream shared weight encoder. The floating image and the fixed image are extracted through four consecutive convolutional blocks. Each convolutional block outputs a floating feature map and a fixed feature map. The floating feature map and the fixed feature map are downsampled through an average pooling layer between adjacent convolutional blocks. Finally, four floating feature maps and four fixed feature maps with different resolutions are obtained.

[0017] Step (3) specifically includes the following steps:

[0018] (3a) Input each of the four floating feature maps with different resolutions and the four fixed feature maps with different resolutions into the 3D related layer in sequence;

[0019] (3b) A four-layer local correlation feature map is generated through a 3D correlation layer. The calculation formula for the 3D correlation layer is as follows:

[0020]

[0021] in, This represents the floating feature map of the i-th layer generated by the improved two-stream shared weight encoder; This represents the fixed feature map of the i-th layer generated by the improved dual-stream shared weight encoder; Represents the floating feature map of the i-th layer. and the fixed feature map of the i-th layer Local correlation feature map generated by 3D correlation layer; k represents the maximum displacement radius; dx, dy, and dz represent the displacement in the x, y, and z directions, respectively; Represents the fixed feature map of the i-th layer The value at coordinates (x, y, z); This represents the floating feature map of the i-th layer. The outermost layer is zero-filled; express The value at coordinates (x+dx, y+dy, z+dz);

[0022] (3c) Connect each floating feature map, each fixed feature map and each local correlation feature map along the channel dimension, and pass them through a 3D convolutional layer with a kernel size of 3×3×3 and a stride of 1 to obtain a four-layer fused feature map.

[0023] Step (4) specifically includes the following steps:

[0024] (4a) Each layer of local correlation feature map is upsampled by a 3D transposed convolutional layer with a kernel size of 2×2×2 and a stride of 2, and then connected with each layer of fused feature map along the channel dimension to obtain a four-layer connected feature map. The resolution of each layer of local correlation feature map and each layer of fused feature map is the same.

[0025] (4b) Construct a context residual enhancement module, which consists of three consecutive convolutional blocks, namely the fifth convolutional block, the sixth convolutional block and the seventh convolutional block. The fifth and sixth convolutional blocks each contain two 3D convolutional layers with a kernel size of 3×3×3 and a stride of 1 and a ReLU activation function. The seventh convolutional block contains one 3D convolutional layer and a ReLU activation function.

[0026] (4c) Input the feature map after the four-layer connection into the context residual enhancement module, and the context residual enhancement module outputs the feature map after the three-layer enhancement.

[0027] Step (5) specifically includes the following steps:

[0028] (5a) Construct a multi-window gated multilayer perceptron, which consists of three parallel window-gated multilayer perceptron branches and a shared registration head. The three parallel window-gated multilayer perceptron branches include a first branch, a second branch, and a third branch. The first branch uses a window size of 3×3×3, the second branch uses a window size of 5×5×5, and the third branch uses a window size of 7×7×7. Each branch consists of a normalization layer, a linear transformation layer, a GELU activation function, and a spatial gate unit. The shared registration head consists of a 3D convolutional layer with a kernel size of 3×3×3 and a stride of 1, and a convolutional layer with 3 output channels.

[0029] (5b) After the first-layer fused feature map is passed through a normalization layer, it is input into a multi-window gated multilayer perceptron. After each branch, the shared registration head generates three velocity fields, which are respectively in, This represents the velocity field generated after the first branch and the shared registration head. This represents the velocity field generated after the second branch and the shared registration head; This represents the velocity field generated after the third branch and the shared registration head;

[0030] For three velocity fields First, the values ​​are added together, then averaged, and finally, after seven identical scaling and squaring operations, the residual deformation field φ1 is obtained. This residual deformation field φ1 is used as the initial total residual deformation field. The formulas for scaling and squaring are as follows:

[0031]

[0032] Among them, v s φ represents the velocity field of the s-th layer; s Describes v over time t∈[0,1] s Integrating, we obtain the deformation field of the s-th layer;

[0033] (5c) After passing through a normalization layer, the enhanced feature map of each layer is input into a multi-window gated multilayer perceptron. After each branch, the shared registration head generates three velocity fields, which are respectively... in, This represents the velocity field generated by the first branch and the shared registration head of the j-th layer. This represents the velocity field generated by the second branch and the shared registration head after passing through the j-th layer; This represents the velocity field generated by the third branch and the shared registration head of the j-th layer;

[0034] For three velocity fields First, the values ​​are added together, then the average is calculated, and finally, the residual deformation fields φ of the second, third, and fourth layers are obtained using formula (2). s s = 2, 3, 4, i.e., φ s Including the second layer residual deformation field φ2, the third layer residual deformation field φ3, and the fourth layer residual deformation field φ4;

[0035] (5d) The initial total residual deformation field, i.e., the residual deformation field φ1, is upsampled and then fused with the second-layer residual deformation field φ2 to obtain the second-layer total residual deformation field φ'2. The second-layer total residual deformation field φ'2 is then upsampled and fused with the third-layer residual deformation field φ3 to obtain the third-layer total residual deformation field φ'3. Finally, the third-layer total residual deformation field φ'3 is fused with the fourth-layer residual deformation field φ4 to obtain the target total residual deformation field φ. The fusion formula is as follows:

[0036]

[0037] Where, φ' s-1 φ represents the total residual deformation field of the (s-1)th layer; s Represents the residual deformation field of the s-th layer; This indicates that the Spatial Transformation Network (STN) will be used to transform φ' s-1 With φ sTwisting; φ represents the total residual deformation field of the target when s = 4; fusion is the fusion operation.

[0038] Step (6) specifically includes the following steps:

[0039] (6a) Obtain two images to be registered, and preprocess the two images to be registered. The preprocessing includes affine transformation, cropping and brightness normalization.

[0040] (6b) One of the preprocessed images to be registered is used as the floating image, and the other preprocessed image to be registered is used as the fixed image. The registration goal is to register the floating image onto the fixed image.

[0041] (6c) Use the target total residual deformation field to register the floating image to obtain a final registered image that is close to the fixed image.

[0042] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: First, the present invention uses a multi-window gated multilayer perceptron to generate the deformation field, which not only considers the key prior knowledge of deformable image registration, but also captures fine-grained long-range dependencies and local correlations at full image resolution. By using a multi-window gated multilayer perceptron to capture sub-deformation fields of different receptive fields, the network can capture scale information and spatial details more precisely, avoiding the error accumulation problem in traditional pyramid registration methods, and has higher computational efficiency, making it suitable for real-time applications. Second, the context residual enhancement module proposed in the present invention further enhances the local correlation and inter-layer correlation of the context. This not only enables the network to capture the correspondence between image pairs that may be missed in the current layer, but also combines the current layer's... Local correlation and the inter-layer correspondence of the previous level provide guidance for the generation of the deformation field of the current layer; third, this invention ensures the smoothness and continuity of deformation by scaling and squaring the velocity field, and adopts a continuous deformation strategy to avoid the discontinuity and consistency problems that may be caused by simply adding the deformation field, thus ensuring the rationality and consistency of the deformation process, and is suitable for handling complex medical image registration tasks; fourth, the experimental results of this invention show that the registration accuracy (Dice coefficient) on different datasets is better than existing methods, especially when dealing with complex deformation and multi-scale deformation, which verifies its effectiveness and superiority in practical applications; compared with existing pyramid registration methods, this invention has the advantages of high accuracy, fewer parameters, low cost, and high computational efficiency. Attached Figure Description

[0043] Figure 1 This is a flowchart of the method of the present invention;

[0044] Figure 2 This is a schematic diagram of the structure of the improved dual-stream shared weight encoder in this invention;

[0045] Figure 3 A slice of a three-dimensional cardiac CT image selected for this invention is used as the image to be registered.

[0046] Figure 4 A slice of a three-dimensional cardiac CT image selected for this invention is used as a fixed image.

[0047] Figure 5 To use the present invention Figure 3 Registration Figure 4 The result image. Detailed Implementation

[0048] like Figure 1 As shown, a large-deformation medical image registration method based on multi-window gated multilayer perceptron and context residual enhancement is presented. The method includes the following sequential steps:

[0049] (1) Obtain a three-dimensional medical image dataset, preprocess it, and divide the preprocessed three-dimensional medical image dataset into floating images and fixed images;

[0050] (2) Input the floating image and the fixed image into the improved dual-stream shared weight encoder for feature extraction to obtain four floating feature maps with different resolutions and four fixed feature maps with different resolutions.

[0051] (3) The four floating feature maps with different resolutions and the four fixed feature maps with different resolutions are used to generate local correlation feature maps through the 3D correlation layer. The local correlation feature maps, floating feature maps and fixed feature maps are then fused to obtain a fused feature map.

[0052] (4) After upsampling, the local related feature map is connected to the fused feature map at the same resolution along the channel dimension, and then input into the context residual enhancement module. The context residual enhancement module outputs the enhanced feature map.

[0053] (5) The feature maps of each layer are enhanced to generate residual deformation fields through a multi-window gated multilayer perceptron. The residual deformation fields are then upsampled and fused with the total residual deformation fields to obtain the target total residual deformation field.

[0054] (6) The image to be registered is distorted by the total residual deformation field of the target to obtain the final registration result. Figure 1 The multi-window gMLP module in the system is a multi-window gated multilayer sensor.

[0055] In step (1), the preprocessing includes affine transformation, cropping, and brightness normalization.

[0056] Step (2) specifically includes the following steps:

[0057] (2a) such as Figure 2 As shown, an improved dual-stream shared weight encoder is constructed, which consists of four consecutive convolutional blocks: the first, second, third, and fourth convolutional blocks. Adjacent convolutional blocks are connected by an average pooling layer. The four convolutional blocks have the same structure, each containing two 3D convolutional layers with a kernel size of 3×3×3 and a stride of 1, as well as a LeakyReLU activation function and an instance normalization layer. Max pooling may lose some detailed information because it only retains the maximum value within the pooling window. Average pooling retains more background information and details by averaging all values ​​within the pooling window, and it can better smooth the feature map during feature extraction, which helps to extract global features.

[0058] (2b) The floating image and the fixed image are respectively input into the improved dual-stream shared weight encoder. The floating image and the fixed image are extracted through four consecutive convolutional blocks. Each convolutional block outputs a floating feature map and a fixed feature map. The floating feature map and the fixed feature map are downsampled through an average pooling layer between adjacent convolutional blocks. Finally, four floating feature maps and four fixed feature maps with different resolutions are obtained.

[0059] Step (3) specifically includes the following steps:

[0060] (3a) Input each of the four floating feature maps with different resolutions and the four fixed feature maps with different resolutions into the 3D related layer in sequence;

[0061] (3b) A four-layer local correlation feature map is generated through a 3D correlation layer. The calculation formula for the 3D correlation layer is as follows:

[0062]

[0063] in, This represents the floating feature map of the i-th layer generated by the improved two-stream shared weight encoder; This represents the fixed feature map of the i-th layer generated by the improved dual-stream shared weight encoder; Represents the floating feature map of the i-th layer. and the fixed feature map of the i-th layer Local correlation feature map generated by 3D correlation layer; k represents the maximum displacement radius; dx, dy, and dz represent the displacement in the x, y, and z directions, respectively; Represents the fixed feature map of the i-th layer The value at coordinates (x, y, z); This represents the floating feature map of the i-th layer. The outermost layer is zero-filled; express The value at coordinates (x+dx, y+dy, z+dz);

[0064] (3c) Connect each floating feature map, each fixed feature map and each local correlation feature map along the channel dimension, and pass them through a 3D convolutional layer with a kernel size of 3×3×3 and a stride of 1 to obtain a four-layer fused feature map.

[0065] Step (4) specifically includes the following steps:

[0066] (4a) Each layer of local correlation feature map is upsampled by a 3D transposed convolutional layer with a kernel size of 2×2×2 and a stride of 2, and then connected with each layer of fused feature map along the channel dimension to obtain a four-layer connected feature map. The resolution of each layer of local correlation feature map and each layer of fused feature map is the same.

[0067] (4b) Construct a context residual enhancement module, which consists of three consecutive convolutional blocks, namely the fifth convolutional block, the sixth convolutional block and the seventh convolutional block. The fifth and sixth convolutional blocks each contain two 3D convolutional layers with a kernel size of 3×3×3 and a stride of 1 and a ReLU activation function. The seventh convolutional block contains one 3D convolutional layer and a ReLU activation function.

[0068] (4c) Input the feature map after the four-layer connection into the context residual enhancement module, and the context residual enhancement module outputs the feature map after the three-layer enhancement.

[0069] Step (5) specifically includes the following steps:

[0070] (5a) Construct a multi-window gated multilayer perceptron, which consists of three parallel window-gated multilayer perceptron branches and a shared registration head. The three parallel window-gated multilayer perceptron branches include a first branch, a second branch, and a third branch. The first branch uses a window size of 3×3×3, the second branch uses a window size of 5×5×5, and the third branch uses a window size of 7×7×7. Each branch consists of a normalization layer, a linear transformation layer, a GELU activation function, and a spatial gate unit. The shared registration head consists of a 3D convolutional layer with a kernel size of 3×3×3 and a stride of 1, and a convolutional layer with 3 output channels.

[0071] (5b) After the first-layer fused feature map is passed through a normalization layer, it is input into a multi-window gated multilayer perceptron. After each branch, the shared registration head generates three velocity fields, which are respectively in, This represents the velocity field generated after the first branch and the shared registration head. This represents the velocity field generated after the second branch and the shared registration head; This represents the velocity field generated after the third branch and the shared registration head;

[0072] For three velocity fields First, the values ​​are added together, then averaged, and finally, after seven identical scaling and squaring operations, the residual deformation field φ1 is obtained. This residual deformation field φ1 is used as the initial total residual deformation field. The formulas for scaling and squaring are as follows:

[0073]

[0074] Among them, v s φ represents the velocity field of the s-th layer; s Describes v over time t∈[0,1] s Integrating, we obtain the deformation field of the s-th layer;

[0075] (5c) After passing through a normalization layer, the enhanced feature map of each layer is input into a multi-window gated multilayer perceptron. After each branch, the shared registration head generates three velocity fields, which are respectively... in, This represents the velocity field generated by the first branch and the shared registration head of the j-th layer. This represents the velocity field generated by the second branch and the shared registration head after passing through the j-th layer; This represents the velocity field generated by the third branch and the shared registration head of the j-th layer;

[0076] For three velocity fields First, the values ​​are added together, then the average is calculated, and finally, the residual deformation fields φ of the second, third, and fourth layers are obtained using formula (2). s s = 2, 3, 4, i.e., φ s Including the second layer residual deformation field φ2, the third layer residual deformation field φ3, and the fourth layer residual deformation field φ4;

[0077] (5d) The initial total residual deformation field, i.e., the residual deformation field φ1, is upsampled and then fused with the second-layer residual deformation field φ2 to obtain the second-layer total residual deformation field φ'2. The second-layer total residual deformation field φ'2 is then upsampled and fused with the third-layer residual deformation field φ3 to obtain the third-layer total residual deformation field φ'3. Finally, the third-layer total residual deformation field φ'3 is fused with the fourth-layer residual deformation field φ4 to obtain the target total residual deformation field φ. The fusion formula is as follows:

[0078]

[0079] Where, φ' s-1 φ represents the total residual deformation field of the (s-1)th layer; s Represents the residual deformation field of the s-th layer; This indicates that the Spatial Transformation Network (STN) will be used to transform φ' s-1 With φ s Twisting; φ represents the total residual deformation field of the target when s = 4; fusion is the fusion operation.

[0080] Step (6) specifically includes the following steps:

[0081] (6a) Obtain two images to be registered, and preprocess the two images to be registered. The preprocessing includes affine transformation, cropping and brightness normalization.

[0082] (6b) One of the preprocessed images to be registered is used as the floating image, and the other preprocessed image to be registered is used as the fixed image. The registration goal is to register the floating image onto the fixed image.

[0083] (6c) Use the target total residual deformation field to register the floating image to obtain a final registered image that is close to the fixed image.

[0084] The total loss for network training in this invention is as follows:

[0085] L total =L sim +λL smooth

[0086] Among them, L total L is the total loss; λ is the weight of the regularization term; L sim For locally normalized cross-correlation loss; L smooth This is the loss for smoothing the deformation field.

[0087] To verify the effectiveness of this invention, the dataset used was derived from the WHS2017 public dataset, containing 60 cardiac CT images. 45 images were selected as the training set, and 15 images as the test set. The registration result (Dice coefficient) of the method proposed in this invention surpasses existing medical image algorithms, improving the registration result to 0.870980, as shown in Table 1. The corr-mlp algorithm is a coarse-to-fine multilayer perceptron for correlation sensing.

[0088] Table 1 Comparison of the present invention and existing technologies on the WHS2017 dataset.

[0089] Registration method Affine registration corr-mlp algorithm groupmorph algorithm This invention Dice coefficient 0.656400 0.844915 0.861995 0.870980

[0090] Figure 2This section details the process of extracting floating and fixed features from each layer using the improved two-stream shared weight encoder. First, the floating and fixed images are used as inputs sequentially, each passing through a first convolutional block. The output feature dimension of the first convolutional block is H×W×D×8. Then, an average pooling layer downsamples the feature map, halving its spatial dimension. The downsampled feature map is then passed through the second convolutional block of the improved two-stream shared weight encoder, with an output dimension of H×W×D×8. Downsampling is performed again using an average pooling layer, further halving the spatial dimension of the feature map. This process is repeated, with the feature map sequentially passed through the third and fourth convolutional blocks of the improved two-stream shared-weight encoder, resulting in output dimensions of [missing dimensions]. and Here, we assume that the original floating image and the fixed image are H×W×D, where H represents the image height, W represents the image width, and D represents the image depth.

[0091] Figure 3 A slice of a three-dimensional cardiac CT image selected in this invention is used as the image to be registered.

[0092] Figure 4 A slice of a three-dimensional cardiac CT image selected in this invention is used as a fixed image.

[0093] Figure 5 To use the present invention Figure 3 Registration Figure 4 The results show that by observing the significant deformation areas marked by the boxes in the figures, it can be seen that the present invention effectively solves the problem of large deformation and exhibits excellent registration effect.

[0094] In summary, this invention uses a multi-window gated multilayer perceptron to capture sub-deformation fields from different receptive fields, enabling the network to capture scale information and spatial details more precisely. This avoids the error accumulation problem in traditional pyramid registration methods and offers higher computational efficiency, making it suitable for real-time applications. The context residual enhancement module proposed in this invention further enhances the local and inter-layer correlations of the context. This not only allows the network to capture the correspondences between image pairs that may be missed in the current layer but also combines the local correlations of the current layer with the inter-layer correspondences of the previous layer to provide guidance for the generation of the deformation field in the current layer. This invention ensures the smoothness and continuity of deformation by scaling and squaring the velocity field, employing a continuous deformation strategy. This avoids the discontinuities and inconsistencies that may arise from simply adding deformation fields, making it suitable for handling complex large-deformation medical image registration tasks.

Claims

1. A method for registering large-deformation medical images based on multi-window gated multilayer perceptron and contextual residual enhancement, characterized in that: The method includes the following steps in sequence: (1) Obtain a three-dimensional medical image dataset, preprocess it, and divide the preprocessed three-dimensional medical image dataset into floating images and fixed images; (2) Input the floating image and the fixed image into the improved dual-stream shared weight encoder for feature extraction to obtain four floating feature maps with different resolutions and four fixed feature maps with different resolutions. (3) The four floating feature maps with different resolutions and the four fixed feature maps with different resolutions are used to generate local correlation feature maps through the 3D correlation layer, and the local correlation feature maps, floating feature maps and fixed feature maps are fused to obtain fused feature maps; (4) After upsampling, the local related feature map is connected to the fused feature map at the same resolution along the channel dimension, and then input into the context residual enhancement module. The context residual enhancement module outputs the enhanced feature map. (5) The feature map of each layer is enhanced to generate a residual deformation field through a multi-window gated multilayer perceptron. The residual deformation field is then upsampled and fused with the total residual deformation field to obtain the target total residual deformation field. (6) The image to be registered is distorted by the total residual deformation field of the target to obtain the final registration result; Step (4) specifically includes the following steps: (4a) Each layer of local correlation feature map is upsampled by a 3D transposed convolutional layer with a kernel size of 2×2×2 and a stride of 2, and then connected with each layer of fused feature map along the channel dimension to obtain a four-layer connected feature map. The resolution of each layer of local correlation feature map and each layer of fused feature map is the same. (4b) Construct a context residual enhancement module, which consists of three consecutive convolutional blocks, namely the fifth convolutional block, the sixth convolutional block and the seventh convolutional block. The fifth and sixth convolutional blocks each contain two 3D convolutional layers with a kernel size of 3×3×3 and a stride of 1 and a ReLU activation function. The seventh convolutional block contains one 3D convolutional layer and a ReLU activation function. (4c) Input the feature map after the four-layer connection into the context residual enhancement module, and the context residual enhancement module outputs the feature map after the three-layer enhancement; Step (5) specifically includes the following steps: (5a) Construct a multi-window gated multilayer perceptron, which consists of three parallel window-gated multilayer perceptron branches and a shared registration head. The three parallel window-gated multilayer perceptron branches include a first branch, a second branch, and a third branch. The first branch uses a window size of 3×3×3, the second branch uses a window size of 5×5×5, and the third branch uses a window size of 7×7×7. Each branch consists of a normalization layer, a linear transformation layer, a GELU activation function, and a spatial gating unit. The shared registration head consists of a 3D convolutional layer with a kernel size of 3×3×3 and a stride of 1, and a convolutional layer with 3 output channels. (5b) After the first-layer fused feature map is passed through a normalization layer, it is input into a multi-window gated multilayer perceptron. After each branch, the shared registration head generates three velocity fields, which are respectively , , ,in, This represents the velocity field generated after the first branch and the shared registration head. This represents the velocity field generated after the second branch and the shared registration head; This represents the velocity field generated after the third branch and the shared registration head; For three velocity fields , , First, the sums are added together, then the average is calculated. Finally, the residual deformation field is obtained by performing seven identical scaling and squaring operations. The residual deformation field As the initial total residual deformation field; the formulas for the scaling and squaring are as follows: , in, Indicates the first The velocity field of the layer; Indicates time On Integrating, we obtain the first... Deformation field of the layer; (5c) After each enhanced feature map is passed through a normalization layer, it is input into a multi-window gated multilayer perceptron. After each branch, the shared registration head generates three velocity fields, which are respectively , , ,in, This represents the velocity field generated by the first branch and the shared registration head of the j-th layer. This represents the velocity field generated by the second branch and the shared registration head after passing through the j-th layer; This represents the velocity field generated by the third branch and the shared registration head of the j-th layer; For three velocity fields , , First, add the results together and then average them. Finally, use formula (2) to obtain the residual deformation fields of the second, third, and fourth layers. s=2, 3, 4, that is Including the second layer of residual deformation field Third layer residual deformation field and the fourth layer residual deformation field ; (5d) The initial total residual deformation field, i.e., the residual deformation field After upsampling, it is compared with the second layer residual deformation field. By merging, the second layer of total residual deformation field is obtained. The total residual deformation of the second layer After upsampling, it is compared with the third layer residual deformation field. By merging, the total residual deformation field of the third layer is obtained. Then the total residual deformation field of the third layer With the fourth layer residual deformation field By fusing the results, the target total residual deformation field is obtained. The fusion formula is as follows: (3), in, This represents the total residual deformation field of the (s-1)th layer; Represents the residual deformation field of the s-th layer; This indicates that using the Spatial Transformation Network (STN) will... and distortion; This represents the total residual deformation field of the target when s=4; For fusion operation.

2. The large-deformation medical image registration method based on multi-window gated multilayer perceptron and contextual residual enhancement according to claim 1, characterized in that: In step (1), the preprocessing includes affine transformation, cropping, and brightness normalization.

3. The large-deformation medical image registration method based on multi-window gated multilayer perceptron and contextual residual enhancement according to claim 1, characterized in that: Step (2) specifically includes the following steps: (2a) Construct an improved dual-stream shared weight encoder, which consists of four consecutive convolutional blocks, namely the first convolutional block, the second convolutional block, the third convolutional block and the fourth convolutional block. Adjacent convolutional blocks are connected by an average pooling layer. The four convolutional blocks have the same structure. Each convolutional block contains two 3D convolutional layers with a kernel size of 3×3×3 and a stride of 1, as well as a LeakyReLU activation function and an instance normalization layer. (2b) The floating image and the fixed image are respectively input into the improved dual-stream shared weight encoder. The floating image and the fixed image are extracted by four consecutive convolutional blocks. Each convolutional block outputs a floating feature map and a fixed feature map. The floating feature map and the fixed feature map are downsampled by an average pooling layer between adjacent convolutional blocks. Finally, four floating feature maps and four fixed feature maps with different resolutions are obtained.

4. The large-deformation medical image registration method based on multi-window gated multilayer perceptron and contextual residual enhancement according to claim 1, characterized in that: Step (3) specifically includes the following steps: (3a) Input each of the four floating feature maps with different resolutions and the four fixed feature maps with different resolutions into the 3D related layer in sequence; (3b) A four-layer local correlation feature map is generated through a 3D correlation layer. The calculation formula for the 3D correlation layer is as follows: , in, This represents the floating feature map of the i-th layer generated by the improved two-stream shared weight encoder; This represents the fixed feature map of the i-th layer generated by the improved dual-stream shared weight encoder; Represents the floating feature map of the i-th layer. and the fixed feature map of the i-th layer Local correlation feature map generated by 3D correlation layer; k represents the maximum displacement radius; dx, dy, and dz represent the displacement in the x, y, and z directions, respectively; Represents the fixed feature map of the i-th layer The value at coordinates (x, y, z); This represents the floating feature map of the i-th layer. The outermost layer is zero-filled; express The value at coordinates (x+dx, y+dy, z+dz); (3c) Connect each floating feature map, each fixed feature map and each local related feature map along the channel dimension, and pass them through a 3D convolutional layer with a kernel size of 3×3×3 and a stride of 1 to obtain a four-layer fused feature map.

5. The large-deformation medical image registration method based on multi-window gated multilayer perceptron and contextual residual enhancement according to claim 1, characterized in that: Step (6) specifically includes the following steps: (6a) Obtain two images to be registered, and preprocess the two images to be registered, the preprocessing including affine transformation, cropping and brightness normalization; (6b) One of the preprocessed images to be registered is used as the floating image, and the other preprocessed image to be registered is used as the fixed image. The registration goal is to register the floating image onto the fixed image. (6c) Use the target total residual deformation field to register the floating image to obtain a registered image that is close to the fixed image.

Citation Information

Patent Citations

  • Medical image registration method based on prediction of multi-level deformation field

    CN113870327A

  • Unsupervised medical image registration method and system based on multiple channels and residual attention mechanism

    CN118314167A