A medical image restoration method and system based on regional self-attention mechanism

Abstract

The application discloses a medical image recovery method and system based on a regional self-attention mechanism, and relates to the technical field of image processing. The medical image recovery method based on the regional self-attention mechanism comprises the following steps: acquiring a medical image, processing the acquired medical image, and obtaining latent code features; dividing the medical image into regions by using a region guidance network, and calculating a segmentation mask; processing the segmentation mask, and calculating an attention mask; and combining the attention mask and the medical image to obtain a synthetic image. According to the regional self-attention mechanism, the medical image can be divided into multiple regions, and the self-attention mechanism is calculated in each region. The complementarity between pixels is realized, potential interference from pixels in dissimilar regions is avoided, the integrity and continuity of the medical image are ensured, and therefore the accuracy and reliability of medical image processing are improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a medical image restoration system and method based on a region self-attention mechanism. Background Technology

[0002] Medical image restoration aims to restore low-quality medical images to high-quality ones, thereby accelerating imaging and assisting doctors in diagnosis.

[0003] With the development of deep learning, researchers have greatly improved the effectiveness of medical image restoration by using network models based on convolutional neural networks (CNNs) and Transformers. Currently, Transformer-based methods have achieved state-of-the-art results in the field of medical image restoration.

[0004] The powerful capabilities of the Transformer stem from its spatial self-attention mechanism, which allows the model to selectively focus on important regions. Currently, most methods utilize two types of spatial self-attention mechanisms: Global Self-Attention (GSA) and Window-based Self-Attention (WSA). Global self-attention calculates the self-attention mechanism across the entire image control, but it is computationally expensive. Window-based self-attention confines the self-attention calculation to a small window, thereby reducing computational costs and achieving efficient self-attention computation. Both global and window-based self-attention have been proven effective in medical image restoration, improving the results.

[0005] However, current self-attention mechanisms have two problems: (1) Both global self-attention and window self-attention coarsely aggregate some image pixels together, allowing these pixels to perform self-attention calculations to achieve information interaction. However, this coarse pixel aggregation may cause dissimilar content to form connections through the self-attention mechanism, thus causing interference between them. (2) Window self-attention mechanisms use fixed window partitioning to divide the entire image into non-overlapping windows, and then perform self-attention calculations within the windows. This fixed region partitioning easily divides regions with continuous image content into different windows, thereby destroying the continuity of image content and hindering the interaction between similar content.

[0006] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0007] In view of the problems in related technologies, this invention proposes a medical image restoration method and system based on regional self-attention mechanism to overcome the above-mentioned technical problems existing in the existing related technologies.

[0008] Therefore, the specific technical solution adopted by the present invention is as follows:

[0009] According to one aspect of the present invention, a medical image restoration method based on a region self-attention mechanism is provided, the method comprising the following steps:

[0010] S1. Acquire medical images and process them to obtain latent code features;

[0011] S2. Use a region guidance network to divide the medical image into regions and calculate the segmentation mask;

[0012] S3. Process the segmentation mask to calculate the attention mask;

[0013] S4. Combine the attention mask with the medical image to obtain a synthetic image.

[0014] Furthermore, acquiring medical images and processing them to obtain latent code features includes the following steps:

[0015] S11. Acquire medical images and input them into a projection through a convolution operation to obtain shallow features;

[0016] S12. Encode the shallow features at several levels to obtain the latent code features.

[0017] Furthermore, the medical image is divided into regions using a region-guided network, and the segmentation mask is calculated, including the following steps:

[0018] S21. Input the medical image into the region guidance network for segmentation processing to obtain a segmented image;

[0019] S22. Calculate the segmentation mask within the segmented image based on the post-processing function.

[0020] Furthermore, the formula for the segmentation mask is:

[0021] M = ostprocess(SAM(I) LQ ))

[0022] In the formula, M represents the segmentation mask;

[0023] Postprocess(·) represents a post-processing function;

[0024] SAM stands for Regional Guidance Network;

[0025] I LQ Represents medical images.

[0026] Further processing of the segmentation mask to calculate the attention mask includes the following steps:

[0027] S31. Using the nearest neighbor interpolation method, the segmentation mask is decoded in several stages to obtain a segmentation mask with the same resolution as the latent code features.

[0028] S32. The downsampled segmentation mask is reshaped and expanded using the Reshape function to calculate the attention mask. Further, the formula for reshaping and expanding the segmentation mask is:

[0029] M′=Reshape((I LQ ))

[0030] In the formula, M′ represents the segmentation mask after shaping and unfolding;

[0031] Reshape(·) represents the shape expansion function;

[0032] NearestInterpolation(·) represents the nearest neighbor interpolation function;

[0033] I LQ Represents medical images.

[0034] Furthermore, the formula for the attention mask is:

[0035]

[0036] In the formula, M ATT Indicates an attention mask;

[0037] λ represents a constant with negative infinity;

[0038] L represents the number of regions;

[0039] i represents the counting variable;

[0040] M i ′ This represents the segmentation mask after shaping;

[0041] T represents the matrix transpose.

[0042] Furthermore, combining the attention mask with the medical image to obtain the synthetic image includes the following steps:

[0043] S41. Combining the region self-attention mechanism and attention mask, perform convolution operation on the medical image to output the projection and obtain the residual image;

[0044] S42. The obtained residual image is summed element-wise with the medical image to obtain the composite image.

[0045] Furthermore, the formula combining the region self-attention mechanism and the attention mask is as follows:

[0046] RSA(Q,K,V)=Softmax(M ATT +QK T / α)V

[0047] In the formula, RSA represents the region self-attention mechanism;

[0048] Q represents the query vector for input latent code features;

[0049] K represents the key vector of the input latent code features;

[0050] V represents the value vector of the input latent code features;

[0051] Softmax(·) represents the activation function;

[0052] M ATT Indicates an attention mask;

[0053] T denotes matrix transpose;

[0054] α represents a learnable adjustment parameter.

[0055] According to another aspect of the present invention, a medical image restoration system based on a region self-attention mechanism is also provided, the medical image restoration system based on a region self-attention mechanism includes an image processing module, an image segmentation module, an attention mask calculation module, and a synthesized image module;

[0056] The image processing module is used to acquire medical images and process the acquired medical images to obtain latent code features;

[0057] The image segmentation module is used to segment medical images into regions using a region guidance network and calculate the segmentation mask.

[0058] The attention mask calculation module is used to process the segmentation mask and calculate the attention mask.

[0059] The image synthesis module combines attention masks with medical images to obtain synthetic images.

[0060] The beneficial effects of this invention are as follows:

[0061] 1. This invention utilizes a region self-attention mechanism, which allows medical images to be divided into multiple regions. Self-attention calculations are performed within each region, achieving not only complementarity between pixels and avoiding potential interference from pixels in dissimilar regions, but also ensuring the integrity and continuity of the medical image, thereby improving the accuracy and reliability of medical image processing.

[0062] 2. This invention calculates an attention mask that can highlight the most important feature information for a specific area in a medical image, thereby helping to enhance the detail and clarity of the medical image. When the attention mask is combined with the synthesized medical image, it can accurately restore and enhance the key information in the image while suppressing the interference of irrelevant information. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart of a medical image restoration method based on a region self-attention mechanism according to an embodiment of the present invention;

[0065] Figure 2 This is a network framework diagram of a medical image restoration method based on a region self-attention mechanism according to an embodiment of the present invention;

[0066] Figure 3 This is a block diagram illustrating the principle of a medical image restoration system based on a region self-attention mechanism according to an embodiment of the present invention.

[0067] Figure 4 This is a comparison diagram of a medical image restoration method based on a region self-attention mechanism according to an embodiment of the present invention in PET image synthesis;

[0068] Figure 5 This is a comparison diagram of a medical image restoration method based on a region self-attention mechanism according to an embodiment of the present invention in CT image denoising.

[0069] Figure 6 This is a comparison image of a medical image restoration method based on a region self-attention mechanism according to an embodiment of the present invention on super-resolution of pathological images.

[0070] In the picture:

[0071] 1. Image processing module; 2. Image segmentation module; 3. Attention mask calculation module; 4. Image synthesis module. Detailed Implementation

[0072] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention.

[0073] According to embodiments of the present invention, a medical image restoration method and system based on a regional self-attention mechanism are provided.

[0074] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figures 1-2 As shown, the medical image restoration method based on region self-attention mechanism according to an embodiment of the present invention includes the following steps:

[0075] S1. Acquire medical images and process them to obtain latent code features.

[0076] Specifically, acquiring medical images and processing them to obtain latent code features includes the following steps:

[0077] S11. Acquire medical images and input them into a projection through a convolution operation to obtain shallow features;

[0078] S12. Encode the shallow features at several levels to obtain the latent code features.

[0079] It should be noted that the medical image is input and projected through a 3×3 convolution to obtain shallow features. These shallow features are then encoded in two levels to obtain low-resolution latent code features.

[0080] S2. Use a region-guided network to divide the medical image into regions and calculate the segmentation mask.

[0081] Specifically, the process of using a region-guided network to segment medical images and calculate the segmentation mask includes the following steps:

[0082] S21. Input the medical image into the region guidance network for segmentation processing to obtain the segmented image.

[0083] It should be noted that the Segment Anything Model (SAM) is used as the region guidance network to achieve the segmentation of image regions. SAM can accurately segment any object in any image, including medical images, without additional training.

[0084] S22. Calculate the segmentation mask within the segmented image based on the post-processing function.

[0085] Specifically, the formula for the segmentation mask is:

[0086] M = ostprocess(SAM(I) LQ ))

[0087] In the formula, M represents the segmentation mask;

[0088] Postprocess(·) represents a post-processing function;

[0089] SAM stands for Regional Guidance Network;

[0090] I LQ Represents medical images.

[0091] It should be further explained that the post-processing operation on the segmented image ensures that the obtained segmentation mask can divide the image with length H and width W into L non-overlapping regions (the number of regions L varies with the image), and ensures that each pixel in the image belongs to a unique region. Specifically, if the same pixel belongs to different segmentation mask regions, it is classified as the mask region with the smallest area; if the same pixel does not belong to any segmentation mask region, it is classified as the background region.

[0092] S3. Process the segmentation mask to calculate the attention mask.

[0093] Specifically, processing the segmentation mask to calculate the attention mask involves the following steps:

[0094] S31. Using the nearest neighbor interpolation method, the segmentation mask is decoded in several stages to obtain a segmentation mask with the same resolution as the latent code features.

[0095] It should be noted that the segmentation mask is decoded in two stages to obtain a segmentation mask with the same resolution as the latent code features.

[0096] Furthermore, the convolutional blocks in encoding and decoding consist of two 3×3 convolutions, and feature downsampling and upsampling use pixel-shuffle and pixel-unshuffle, respectively; in order to assist the image restoration process, the features of each encoder are summed with the output features of the corresponding decoder through skip connections.

[0097] S32. The downsampled segmentation mask is reshaped and expanded using the Reshape function to calculate the attention mask. Specifically, the formula for reshaping and expanding the segmentation mask is:

[0098] M′=Reshape((I LQ ))

[0099] In the formula, M′ represents the segmentation mask after shaping and unfolding;

[0100] Reshape(·) represents the shape expansion function;

[0101] NearestInterpolation(·) represents the nearest neighbor interpolation function;

[0102] I LQ Represents medical images.

[0103] Specifically, the formula for the attention mask is:

[0104]

[0105] In the formula, M aAT Indicates an attention mask;

[0106] λ represents a constant with negative infinity;

[0107] L represents the number of regions;

[0108] i represents the counting variable;

[0109] M i ′ This represents the segmentation mask after shaping;

[0110] T represents the matrix transpose.

[0111] S4. Combine the attention mask with the medical image to obtain a synthetic image.

[0112] Specifically, combining attention masks with medical images to obtain synthetic images includes the following steps:

[0113] S41. Combining the region self-attention mechanism and attention mask, the medical image is convolved and projected to obtain the residual image.

[0114] Specifically, the formula combining the region self-attention mechanism and the attention mask is as follows:

[0115] RSA(Q,K,V)=Softmax(M ATT +QK T / α)V

[0116] In the formula, RSA represents the region self-attention mechanism;

[0117] Q represents the query vector for input latent code features;

[0118] K represents the key vector of the input latent code features;

[0119] V represents the value vector of the input latent code features;

[0120] Softmax(·) represents the activation function;

[0121] M aTT Indicates an attention mask;

[0122] T denotes matrix transpose;

[0123] α represents a learnable adjustment parameter.

[0124] It should be added that by combining the region self-attention mechanism with the attention mask, the correlation between pixels from different regions approaches 0, thereby limiting the calculation of self-attention to each region.

[0125] S42. The obtained residual image is summed element-wise with the medical image to obtain the composite image.

[0126] It should be noted that the formula for calculating the composite image is:

[0127]

[0128] In the formula, Indicates a composite image;

[0129] I LQ Represents medical images;

[0130] I R This represents the residual image.

[0131] It should be further explained that using the 1-norm loss to constrain the restoration of the basic image content, minimizing the loss function means minimizing the image restored by the network. and high-quality images HQ Differences between them:

[0132]

[0133] In the formula, Loss represents minimizing the image recovered by the network. and high-quality images HQ The difference between them;

[0134] This indicates minimizing the network to recover the image;

[0135] I HQ Indicates a high-quality image;

[0136] |·|1 represents the 1-norm loss.

[0137] In addition, the number of convolutional blocks, the region self-attention mechanism, and the attention mask were set to N1=2=4=5=2 and N3=12, respectively; the number of channels of the shallow features obtained from the input projection was C=64; and the trained SAM model based on the ViT-B backbone network was used to perform region segmentation on the input medical image.

[0138] For SAM model training, it was implemented using PyTorch on a workstation equipped with an NVIDIA A100 GPU; the Adam optimizer was used, the learning rate was fixed at 0.0001, the training data block size was 128×128, the training batch size was set to 8, and a total of 200,000 iterations were performed.

[0139] By applying the methods described above in the embodiments, it surpasses the most advanced algorithms currently available in PET image synthesis, CT image denoising, and pathological image super-resolution, as detailed below: PET image synthesis results are as follows... Figure 4 As shown, this method surpasses ARGAN, the most advanced algorithm for 2D PET image restoration; the CT denoising effect is as follows: Figure 5 As shown, this method surpasses the current state-of-the-art CT denoising algorithm, CTformer; the pathological super-resolution results are as follows. Figure 6 As shown, this method surpasses the state-of-the-art algorithm, SwinIR.

[0140] like Figure 2 As shown, according to another embodiment of the present invention, a medical image restoration system based on a region self-attention mechanism is also provided. The medical image restoration system based on a region self-attention mechanism includes an image processing module 1, an image segmentation module 2, an attention mask calculation module 3, and a composite image module 4.

[0141] Image processing module 1 is used to acquire medical images and process the acquired medical images to obtain latent code features;

[0142] Image segmentation module 2 is used to segment medical images into regions using a region guidance network and calculate a segmentation mask.

[0143] Attention mask calculation module 3 is used to process the segmentation mask and calculate the attention mask;

[0144] The image synthesis module 4 is used to combine the attention mask with the medical image to obtain the synthesized image.

[0145] In summary, by utilizing the technical solution of this invention, medical images can be divided into multiple regions through a region self-attention mechanism. The self-attention mechanism is calculated within each region, achieving not only pixel complementarity and avoiding potential interference from pixels in dissimilar regions, but also ensuring the integrity and continuity of the medical image, thereby improving the accuracy and reliability of medical image processing. By calculating the attention mask, the most important feature information for specific regions in the medical image can be highlighted, thus helping to enhance the detail and clarity of the medical image. When the attention mask is combined with the synthesized medical image, key information in the image can be accurately recovered and enhanced while suppressing interference from irrelevant information.

[0146] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A medical image restoration method based on a region self-attention mechanism, characterized in that, This medical image restoration method based on regional self-attention mechanism includes the following steps: S1. Acquire medical images and process them to obtain latent code features; S2. Use a region guidance network to divide the medical image into regions and calculate the segmentation mask; S3. Process the segmentation mask to calculate the attention mask; S4. Combine the attention mask with the medical image to obtain the synthesized image; S3 includes the following steps: S31. Using the nearest neighbor interpolation method, the segmentation mask is decoded in several stages to obtain a segmentation mask with the same resolution as the latent code features. S32. Use the Reshape function to shape and expand the downsampled segmentation mask to calculate the attention mask; The formula for shaping and unfolding the segmentation mask is: In the formula, This represents the segmentation mask after shaping and unfolding. Represents the shape expansion function; This represents the nearest neighbor interpolation function; Represents medical images; The formula for the attention mask is: In the formula, Indicates an attention mask; A constant representing negative infinity; Indicates the number of regions; Represents a counting variable; This represents the shaped segmentation mask; T represents the matrix transpose. S4 includes the following steps: S41. Combining the region self-attention mechanism and attention mask, perform convolution operation on the medical image to output the projection and obtain the residual image; S42. The obtained residual image is summed element-wise with the medical image to obtain the composite image; The formula combining the region self-attention mechanism and the attention mask is as follows: In the formula, RSA represents the region self-attention mechanism; Q represents the query vector of the input latent code features; K represents the key vector of the input latent code features; and V represents the value vector of the input latent code features. Indicates the activation function; This represents the attention mask; T represents the matrix transpose. This represents the learnable adjustable parameters.

2. The medical image restoration method based on a region self-attention mechanism according to claim 1, characterized in that, The process of acquiring medical images and processing them to obtain latent code features includes the following steps: S11. Acquire medical images and input them into a projection through a convolution operation to obtain shallow features; S12. Encode the shallow features at several levels to obtain the latent code features.

3. The medical image restoration method based on a region self-attention mechanism according to claim 1, characterized in that, The process of using a region-guided network to segment medical images and calculate a segmentation mask includes the following steps: S21. Input the medical image into the region guidance network for segmentation processing to obtain a segmented image; S22. Calculate the segmentation mask within the segmented image based on the post-processing function.

4. The medical image restoration method based on regional self-attention mechanism according to claim 3, characterized in that, The formula for the segmentation mask is: In the formula, M represents the segmentation mask; Indicates the post-processing function; SAM stands for Regional Guidance Network; Represents medical images.

5. A medical image restoration system based on a region self-attention mechanism, used to implement the medical image restoration method based on a region self-attention mechanism as described in any one of claims 1-4, characterized in that, The medical image restoration system based on the region self-attention mechanism includes an image processing module, an image segmentation module, an attention mask calculation module, and a synthesized image module; The image processing module is used to acquire medical images and process the acquired medical images to obtain latent code features; The image segmentation module is used to segment medical images into regions using a region guidance network and calculate a segmentation mask. The attention mask calculation module is used to process the segmentation mask and calculate the attention mask. The image synthesis module is used to combine an attention mask with a medical image to obtain a synthesized image.

Images