A self-attention based local cross-correlation enhanced cardiac registration method

Abstract

The present application relates to the cross technical field of deep learning and medical image registration, and disclose a kind of local cross-correlation enhanced heart registration method based on self-attention, comprising the following steps: S1, image acquisition: obtain heart MR image, PET image as sample data, while the heart of MR, PET image is labeled;S2, image pre-processing: by resampling, cropping, normalization and denoising to image processing;S3, model construction;S4, model training;S5, model evaluation and test;The present application combines CNN and Transformer, realizes heart image registration by UNet structure and self-attention mechanism, pays attention to details and overall structure, improves adaptability, adopts semi-supervised learning to combine segmentation label, improves registration quality, enhances the modeling of local context information, by setting smooth constraint function, reduce folding structure and not in line with biological results, ensure the topological characteristics of heart image deformation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and medical image registration, in particular to a local cross-correlation enhanced heart registration method based on self-attention. BACKGROUND

[0002] According to the latest statistics of the World Health Organization, cardiovascular diseases have posed a threat to human health. In China, the prevalence and mortality rates of CVD are on the rise, and the total number of patients has exceeded 500 million, which is more than other diseases such as tumors. CVD accounts for more than 40% of the composition of resident disease deaths, meaning that in every five cases of death, two are caused by cardiovascular diseases. The heart, as a key organ for maintaining life, its internal structure changes, such as myocardial hypertrophy and cardiac chamber enlargement, will significantly increase the burden on the heart, thus posing a threat to life. Therefore, in-depth study of the physiological and pathological characteristics of the heart and early non-invasive diagnosis are crucial for the prevention and treatment of cardiovascular diseases. MRI and PET have been widely used in the detection and diagnosis of heart diseases. MRI can provide high-resolution structural information of the heart, and PET can realize quantitative evaluation of myocardial activity, blood perfusion and metabolism. Registering images of the two different modalities is an important task in the field of heart medical analysis. By integrating different modalities of images on a unified reference, more rich semantic information of images can be fused, and a more comprehensive understanding of the patient's disease can be achieved.

[0003] The neural network structure used in the existing heart registration algorithm is mostly a convolutional neural network, but the local receptive field of the CNN limits its ability to capture long-range dependencies. The registration of the heart structure needs to consider the relative position relationship of the ventricular wall, the heart cavity and the heart with other organs. These features often involve global spatial information, which is difficult for standard CNN to capture. At the same time, the heart is constantly moving and changing during normal physiological processes, so the use of CNN is not enough to handle the dynamic changes of the heart during the beating cycle. The dynamic nature of the heart requires the model to capture the deformation pattern over time, which requires the network to process multi-modal data and understand its temporal dependence. Moreover, most of the existing technologies are supervised learning or unsupervised learning. For the heart registration task, supervised learning requires a large amount of data labeling, i.e., the registration of the deformation parameters. This labeling is time-consuming and expensive. Unsupervised learning does not require any labeling, but this method faces the problem of long time and low efficiency in the process of heart feature learning. In addition, due to the lack of labeling information, the results of unsupervised learning are difficult to quantify and evaluate. Moreover, the existing registration method cannot maintain the original differential characteristics and topological features of the heart image during the transformation process, which may cause the registration image to appear unreasonable folding, which does not conform to the actual biomechanical characteristics of the human body organs. The gradient of the displacement field used as a smoothing constraint in the existing technology can improve the smoothness of the registration, but this method applies the same restriction to all deformations in the image without distinguishing between large and small deformation regions, resulting in too strict deformation constraints for large displacement regions, which cannot adapt to the actual changes required. The contraction and expansion process of the heart organ will produce a large deformation. SUMMARY

[0004] The purpose of the present application is to provide a local cross-correlation enhanced heart registration method based on self-attention to solve the problems raised in the above background.

[0005] To achieve the above purpose, the present application provides the following technical solution: a local cross-correlation enhanced heart registration method based on self-attention, comprising the following steps:

[0006] S1, image acquisition: acquiring heart MR images and PET images as sample data, and labeling the heart of the MR and PET images;

[0007] S2, image preprocessing: processing the images through resampling, cropping, normalization and denoising;

[0008] S3, model construction: combining the model structure of the convolutional network CNN and the transformer;

[0009] S4, model training: the training method uses semi-supervised learning, and uses the labels of fixed images and moving images to assist in training the neural network;

[0010] S5. Model Evaluation and Testing: Evaluate the model's performance on an independent test set and adjust the model's parameters.

[0011] Preferably, in step S1, the heart in MR and PET images needs to be labeled to obtain corresponding heart labels, and then the labeled sample data is divided into training set and test set.

[0012] Preferably, in S2, resampling refers to unifying the resolution by interpolating to a resolution of 1mm×1mm×1mm using B-spline. In S2, cropping refers to cropping the cardiac image to the region of interest, and the size of the cropped image is 256x256 pixels.

[0013] Preferably, in S2, normalization refers to normalizing the image intensity to the range [0,1], and in S2, denoising refers to denoising the image using median filtering.

[0014] Preferably, in S3, the network model includes 4 convolutional blocks, 1 visual coding block, 4 upsampling blocks and 1 generation block. The network model takes cardiac MR images and cardiac PET images as inputs, which are respectively used as fixed images and moving images. The output of the network is a displacement field, which is applied to the moving image. The network model as a whole uses a UNet encoder-decoder structure.

[0015] Preferably, the decoder can combine cardiac image information with location information, and the intermediate skip connections combine the deep features of the encoder with the shallow features of the decoder. The structure of the convolutional block is as follows: a convolutional layer is connected to a leakyReLU layer and an instance normalization layer, then another convolutional layer is connected to a leakyReLU layer and an instance normalization layer, and finally a max pooling layer is connected. The max pooling layer is used to reduce the spatial dimension of the feature map.

[0016] Preferably, the visual coding block consists of a segmentation layer, an embedding layer, 12 transformers, and a convolutional block. The upsampling block includes an upsampling layer, a stitching layer, a convolutional layer, LeaklyReLU, and instance normalization.

[0017] Preferably, the convolutional layer can extract local features from cardiac images, the LeakyReLU activation function provides a non-linear mapping, and the instance normalization normalizes each channel of each input image.

[0018] Preferably, the convolutional block inputs the feature map into the segmentation layer. Before inputting the feature map, it needs to be segmented and embedded. The segmentation layer divides the feature map into multiple small blocks, which are then input into the embedding layer. Each small block is 8×8 pixels in size. The embedding layer has a linear mapping layer and a position encoding layer. The linear mapping layer uses 1D convolution, and the position encoding layer... pos It is a learnable parameter matrix of size N×D, where N is the number of image patches and D is the dimension, i.e., 252 dimensions. The final representation of a patch is obtained by adding its embedding vector v to its corresponding positional encoding: v = v0 + E pos v0 is the embedding obtained through the linear mapping layer. The vector obtained through linear mapping and positional encoding is fed into the transformer module for deep feature extraction and transformation. The transformer module consists of a self-attention mechanism and layer normalization.

[0019] Preferably, in step S4, labels of fixed and moving images are used to assist in training the neural network. The number of training epochs is set to 1000, the AdamW optimization algorithm is used, the initial learning rate is set to 0.0001, and the ReduceLROnPlateau learning rate is used for automatic decay. The patience parameter is set to 30 times. When the loss does not decrease in 30 consecutive epochs, the learning rate is automatically reduced to 1 / 10 of the original.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] 1. This invention sets up a network model that combines CNN with Transformer. The CNN uses a UNet structure, and the combination with Transformer allows the model to simultaneously focus on the details and overall structure of the heart image, thereby understanding the complex deformation of the heart. At the same time, the self-attention mechanism of Transformer can capture the correlation between different regions over long distances, helping the model learn the complex movements of the heart during the beating cycle. Combining the two can also provide the network with better adaptability to cope with the variability of heart structure in different patients.

[0022] 2. This invention utilizes semi-supervised learning in the network model, leveraging segmentation labels from cardiac images to provide prior information about the heart, guiding the model's registration process and improving registration quality. Simultaneously, the segmentation labels provide precise information about different anatomical structures in the cardiac images, helping the model align cardiac image information before and after registration, enhancing the model's modeling of local contextual information, and improving its understanding of local cardiac features. Furthermore, the segmentation labels are used to calculate the final loss and can help the model generate a smoother deformation field, making the structure more anatomically consistent.

[0023] 3. This invention obtains a smooth displacement field by setting a smooth constraint function based on the displacement field, pooling the displacement field, and calculating the L2-norm regularization loss between the displacement field and the pooled displacement field. The new smooth constraint loss can ensure the topological features of cardiac images during deformation, and reduce the generation of folded structures and biologically inconsistent results by the model. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of a preferred embodiment of the self-attention-based local cross-correlation-enhanced cardiac registration method provided by the present invention.

[0025] Figure 2 This is a schematic diagram of the model construction structure provided by the present invention;

[0026] Figure 3 This is a schematic diagram of the convolution block structure provided by the present invention;

[0027] Figure 4 This is a schematic diagram of the visual coding block structure provided by the present invention;

[0028] Figure 5 This is a schematic diagram of the upsampling block structure provided by the present invention;

[0029] Figure 6 This is a schematic diagram of the model training process provided by the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Please see Figures 1-6 As shown, a method for enhancing cardiac registration based on self-attention and local cross-correlation includes the following steps:

[0032] S1. Image Acquisition: Acquire cardiac MR images and PET images as sample data, and simultaneously annotate the cardiac images in MR and PET.

[0033] S2. Image preprocessing: Image processing is performed through resampling, cropping, normalization, and noise reduction.

[0034] S3. Model Construction: A model structure combining Convolutional Networks (CNN) and Transformers;

[0035] S4. Model Training: The training method uses semi-supervised learning, using labels from fixed and moving images to assist in training the neural network;

[0036] S5. Model Evaluation and Testing: Evaluate the model's performance on an independent test set and adjust the model's parameters.

[0037] In S1, the heart in MR and PET images needs to be labeled to obtain the corresponding heart labels, and then the labeled sample data is divided into training set and test set.

[0038] In S2, resampling refers to standardizing the resolution by interpolating to a resolution of 1mm×1mm×1mm using B-spline interpolation. Cropping in S2 refers to cropping the cardiac image to the region of interest, resulting in a cropped image size of 256x256 pixels. Cropping reduces computational load and allows focus to be placed on cardiac image registration.

[0039] In S2, normalization refers to normalizing the image intensity to the range [0,1]. In S2, denoising refers to using median filtering to remove noise from the image. The beating of the heart introduces motion artifacts, and breathing introduces noise into the image. Therefore, denoising can improve the accuracy of the image.

[0040] In S3, the network model contains four convolutional blocks, one visual encoding block, four upsampling blocks, and one generator block. The network takes cardiac MR and PET images as input, representing stationary and moving images respectively. The output is a displacement field, which is applied to the moving image. The network model uses a UNet encoder-decoder structure. By setting stationary and moving images, spatial transformations can be performed on the moving image to achieve cardiac image registration. The UNet encoder-decoder structure allows the encoder to fully capture the contextual information of the cardiac images.

[0041] The decoder combines cardiac image information with location information. Intermediate skip connections combine deep features from the encoder with shallow features from the decoder. The convolutional block structure consists of one convolutional layer connected to a leaky ReLU layer and an instance normalization layer, followed by another convolutional layer connected to another leaky ReLU layer and an instance normalization layer, and finally a max-pooling layer to reduce the spatial dimensionality of the feature maps. The decoder helps the model fuse features at different scales, improving registration accuracy.

[0042] The visual encoding block consists of a segmentation layer, an embedding layer, 12 transformers, and a convolutional block. The upsampling block includes an upsampling layer, a concatenation layer, a convolutional layer, LeakyReLU, and instance normalization. The upsampling layer upsamples the low-resolution feature maps to a higher resolution, learning complex feature maps. The concatenation layer combines the outputs of the preceding convolutional blocks, integrating deep and shallow features to help the model fuse features at different scales. The final generation block uses a convolutional network to generate a displacement field for subsequent heart registration.

[0043] Convolutional layers can extract local features from cardiac images, the LeakyReLU activation function provides a non-linear mapping, and instance normalization normalizes each channel of each input image. Through the action of convolutional layers, the model can understand the structure of the heart, the LeakyReLU activation function helps the network learn more complex feature representations, and instance normalization can reduce internal covariate shifts and accelerate the training process.

[0044] The convolutional block inputs the feature map into the segmentation layer. Before entering the segmentation layer, the feature map needs to be segmented and embedded. The segmentation layer divides the feature map into multiple small blocks, which are then input into the embedding layer. Each small block is 8×8 pixels in size. The embedding layer consists of a linear mapping layer and a position encoding layer. The linear mapping layer uses 1D convolution, and the position encoding layer uses E... pos It is a learnable parameter matrix of size N×D, where N is the number of image patches and D is the dimension, i.e., 252 dimensions. The final representation of a patch is obtained by adding its embedding vector v to its corresponding position code: v = υ0 + E posThe embedding v0 is obtained through a linear mapping layer. The vector obtained after linear mapping and positional encoding is fed into the transformer module for deep feature extraction and transformation. The transformer module consists of a self-attention mechanism and layer normalization. The upsampling block contains an upsampling layer, a concatenation layer, a convolutional layer, LeakyReLU, and IN. The current mapping layer uses 1D convolution, which can share weights across the entire vector, reducing model parameters. The positional encoding layer uses the positional information of each element in the source sequence. The attention mechanism allows the model to focus on different regions in the cardiac image while processing the representation of each image patch, helping the model capture the complex relationships between different locations in the cardiac image, such as changes in shape, size, and relative position. Layer normalization reduces the internal variance between different training samples, making training smoother. Using the transformer structure also allows for efficient parallel processing, improving the model's training speed. Its calculation formula is z′. l =MSA(LN(z) l-1 ))+z l-1 and z l =MLP(LN(z′) l ))+z′ l .

[0045] In S4, labels from both fixed and moving images are used to assist in training the neural network. The number of training epochs is set to 1000, using the AdamW optimization algorithm with an initial learning rate of 0.0001. ReduceLROnPlateau automatic learning rate decay is employed, and the patience parameter is set to 30. If the loss does not decrease for 30 consecutive epochs, the learning rate is automatically reduced to 1 / 10 of its original value. F and M represent the fixed and moving images in the above figure, respectively, and are the same size. The fixed image F and the moving image M are first input into the network model. The network output is a displacement field y, which aligns the fixed image F and the moving image M in the field of view, thus completing the heart registration. For training a neural network, we can consider y as a function of the input images F, M and the learnable model parameters θ, i.e., y(x) = y(θ; F, M, x). During training, the network parameters θ are learned to minimize the displacement field y using the loss function Loss. The overall loss formula for the neural network is as follows: L(F, M, bF, bM, y) = δ·D(F, M(y)) + α·Dfs(y) + β·B(bF, bM(y)). The formula consists of three parts: two distance losses, D and B, and a smoothing constraint function Dfs(Displacementfieldsmoothingloss, Dfsloss) based on the displacement field. The distance metric D quantifies the similarity between the fixed image F and the deformed moving image bM(y), and Dfs constrains the smoothness of the displacement field. The second distance metric B quantifies the similarity between the fixed image label bF and the deformed image label bM(y). The parameters δ, α, β ≥ 0 are the weight factors of D, Dfs, and B, respectively. Furthermore, the formula for the distance metric D is as follows:

[0046]

[0047] Where <·, ·> denote the dot product of two vectors. and These are the gradients of the deformed image and the fixed image at point x, respectively, where x is the position of a pixel in the image, Ω is the image domain, therefore x ∈ Ω, ∈ is the edge parameter, and ∈ > 0. DFS is a smoothing constraint function based on the displacement field, which can provide different smoothing constraints for each pixel in the cardiac image. The deformation magnitude of any point x in the image should be approximately the deformation magnitude of the voxels surrounding x. Therefore, the average displacement of points around point x can be used as the true displacement of point x and used for supervision during training. The method is to perform average pooling on the network model output y. This invention uses average pooling with a pooling kernel size of 3 and a stride of 1. Assuming the size of the original displacement field is (H, W), the size after pooling is (H-2, W-2), which is inconsistent with the size of the fixed image. However, in general, the edges of the image are background and do not contain meaningful information. Therefore, the outermost layer of the displacement field has no displacement or minimal displacement, so the outermost layer of the displacement field has little impact on smoothing. Therefore, 0 can be used to fill the displacement field y, and then the L2 norm is calculated. The specific formula is as follows.

[0048]

[0049] Finally, the distance loss B between the fixed image label and the deformed image label is calculated as follows:

[0050]

[0051] Working principle: First, cardiac MR and PET images are used as sample data, and the cardiac regions in these images are labeled to generate corresponding cardiac labels. The labeled sample data is divided into training and testing sets for subsequent model training and validation. B-spline interpolation is used to unify the image resolution to 1mm×1mm×1mm, and the cardiac images are cropped to the region of interest to reduce computation and focus on cardiac image registration. The cropped image size is 256x256 pixels. Image intensity is normalized to the range [0,1] to reduce intensity differences between different images. Median filtering is used to denoise the images and improve image accuracy. Then, a convolutional network (CNN) and transformer model structure are combined. The network model contains 4 convolutional blocks, 1 visual encoding block, 4 upsampling blocks, and 1 generator block. The network input is cardiac MR and cardiac PET images, used as stationary and moving images respectively. The network output is a displacement field, which is applied to the moving image to perform spatial... Transformation is used to register cardiac images. Before the feature map is input, it needs to be segmented and embedded. The segmentation layer divides the feature map into multiple small blocks, which are then input into the embedding layer. The embedding layer has a linear mapping layer and a position encoding layer. The linear mapping layer uses 1D convolution to map the image blocks to a higher-dimensional space. Using a 1D convolution layer can share weights across the entire vector, reducing the model's parameters. The small blocks need to be divided into 8×8 pixel blocks, and then the 1D convolution is used to map the dimension to 252. Then the model is connected to the position encoding layer. Since the Transformer model itself does not have position information, a position encoding layer is introduced to add position information for each image block. The position encoding layer Epos is a learnable parameter matrix of size N×D, where N is the number of image blocks and D is the dimension, i.e., 252 dimensions. The embedding vector v0 of each block is obtained by passing it through the linear mapping layer and adding it to the corresponding position encoding Epos to obtain the final representation v of the block, i.e., v = v0 + Epos. pos The model inputs small, embedded blocks into a Transformer model, which uses a self-attention mechanism to capture the complex relationships between different regions in cardiac images. Through structures such as upsampling blocks and stitching layers, features at different scales are fused. Finally, a displacement field is generated using the generated blocks for cardiac image registration. A semi-supervised learning method is used, combining labels from both fixed and moving images for training. The AdamW optimization algorithm and the ReduceLROnPlateau automatic learning rate decay strategy are employed to optimize model performance. After training, the model's performance is evaluated using an independent test set, and the model parameters are further adjusted. When image analysis of a patient's cardiac disease is required, the corresponding processed images are input into the model to obtain the registration results.

[0052] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0053] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for enhancing cardiac registration based on self-attention and local cross-correlation, characterized in that, Includes the following steps: S1. Image Acquisition: Acquire cardiac MR images and PET images as sample data, and simultaneously annotate the cardiac images in MR and PET. S2. Image preprocessing: Image processing is performed through resampling, cropping, normalization, and noise reduction. S3. Model Construction: A model structure combining Convolutional Networks (CNN) and Transformers; S4. Model Training: The training method uses semi-supervised learning, using labels from fixed and moving images to assist in training the neural network; S5. Model Evaluation and Testing: Evaluate the model's performance on an independent test set and adjust the model's parameters. In step S1, the heart in MR and PET images needs to be labeled to obtain corresponding heart labels, and then the labeled sample data is divided into training set and test set. In S2, resampling refers to unifying the resolution by interpolating to a resolution of 1mm×1mm×1mm using B-spline. In S2, cropping refers to cropping the cardiac image to the region of interest, and the cropped image size is 256x256 pixels. In S2, normalization refers to normalizing the image intensity to the range [0,1], and in S2, denoising refers to using median filtering to denoise the image. In S3, the network model contains 4 convolutional blocks, 1 visual coding block, 4 upsampling blocks and 1 generator block. The network model takes cardiac MR images and cardiac PET images as inputs, which are used as fixed images and moving images, respectively. The output of the network is a displacement field, which is applied to the moving image. The network model as a whole uses a UNet encoder-decoder structure. The decoder can combine cardiac image information with location information. The intermediate skip connections combine the deep features of the encoder with the shallow features of the decoder. The structure of the convolutional block is a convolutional layer connected to a leakyReLU layer and an instance normalization layer, then another convolutional layer connected to a leakyReLU layer and an instance normalization layer, and finally a max pooling layer. The max pooling layer is used to reduce the spatial dimension of the feature map. The visual coding block consists of a segmentation layer, an embedding layer, 12 transformers, and a convolutional block. The upsampling block includes an upsampling layer, a concatenation layer, a convolutional layer, LeaklyReLU, and instance normalization. The convolutional block inputs the feature map into the segmentation layer. Before entering the segmentation layer, the feature map needs to be segmented and embedded. The segmentation layer divides the feature map into multiple small blocks, which are then input into the embedding layer. Each small block is 8×8 pixels in size. The embedding layer has a linear mapping layer and a position encoding layer. The linear mapping layer uses 1D convolution, and the position encoding layer... pos It is a learnable parameter matrix of size N×D, where N is the number of image patches and D is the dimension, i.e., 252 dimensions. The final representation of a patch is obtained by adding its embedding vector v to its corresponding positional encoding: v = v0 + E pos v0 is the embedding obtained through the linear mapping layer. The vector obtained after linear mapping and positional encoding is fed into the transformer module for deep feature extraction and transformation. The transformer module consists of a self-attention mechanism and layer normalization. In step S4, during the training process of the neural network, the formula for calculating the overall loss function Loss of the neural network is as follows: L(F,M,bF,bM,y)=δ·D(F,M(y))+α·Dfs(y)+β·B(bF,bM(y)); F and M represent the fixed image and the moving image, respectively. D and B are both distance losses. D is used to quantify the similarity between the fixed image F and the deformed moving image M(y). B is used to quantify the similarity between the fixed image label bF and the deformed image label bM(y). Dfs is a smooth constraint function based on the displacement field. The parameters δ, α, β≥0 are the weight factors of D, Dfs and B, respectively. The formula for calculating the distance metric D is as follows: ； <·, ·> represent the dot product of two vectors. and These are the gradients of the deformed and fixed images at point x, respectively, where x is the position of a pixel in the image, and Ω is the image domain, x∈Ω. These are edge parameters. >0; The formula for calculating the smooth constraint function Dfs based on the displacement field is as follows: ； The formula for calculating the distance loss B between fixed image labels and deformed image labels is: 。 2. The method for enhancing cardiac registration based on self-attention and local cross-correlation according to claim 1, characterized in that: The convolutional layer can extract local features from cardiac images, the LeakyReLU activation function provides a non-linear mapping, and the instance normalization normalizes each channel of each input image.

3. The method for enhancing cardiac registration based on self-attention local cross-correlation according to claim 1, characterized in that: In step S4, labels for both fixed and moving images are used to assist in training the neural network. The number of training epochs is set to 1000, and the AdamW optimization algorithm is used. The initial learning rate is set to 0.0001, and the ReduceLROnPlateau learning rate is used for automatic decay. The patience parameter is set to 30 times. When the loss does not decrease in 30 consecutive epochs, the learning rate is automatically reduced to 1 / 10 of its original value.

Images