Medical image super-resolution method and system based on feature compensation attention

Abstract

The invention provides a medical image super-resolution method and system based on feature compensation attention, and relates to the field of medical image processing, and the method comprises the steps: building a medical image super-resolution generative adversarial network model which comprises a generator and a discriminator, the generator comprises a first feature extraction component, a feature injection component, a second feature extraction component and a reconstruction component, the outputs of the first feature extraction component and the feature injection component are the input of the second feature extraction component, and the output of the second feature extraction component is the input of the reconstruction component; training the medical image super-resolution generative adversarial network model to obtain a medical image super-resolution model which comprises a generator of the trained medical image super-resolution generative adversarial network model; the high-resolution medical image is obtained based on the low-resolution medical image through the medical image super-resolution model, and the method has the advantage that the high-resolution medical image closer to a real image is generated.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a medical image super-resolution method and system based on feature compensation attention. Background Art

[0002] The development of medical imaging technology has always been an important research direction in the medical field. Among these technologies, CT scanning provides high resolution, multi-planar imaging, rapid acquisition, excellent contrast resolution, and 3D reconstruction capabilities, making it invaluable in detailed anatomical assessment and emergency diagnosis. Magnetic resonance imaging (MRI) plays a vital role in clinical diagnosis and treatment decisions with its advantages of radiation-free and effective soft tissue imaging. In a typical MRI scan, a voxel can represent up to one cubic millimeter of tissue, which means relatively low resolution, hindering accurate diagnosis and treatment. In order to improve the accuracy of diagnosis and treatment, it is increasingly important to improve the resolution of MRI (Magnetic Resonance Imaging) and CT (Computed Tomography) scans.

[0003] High-resolution (HR) and isotropic medical images provide detailed texture and structural information necessary for accurate diagnosis and quantitative analysis. However, it is challenging to obtain such images in clinical practice. The quality of medical images is limited by many factors, including hardware conditions, reconstruction algorithms, acquisition time, physiological movement, and acceptable radiation dose for patients. Clear MRI scans can significantly improve the accuracy of clinical diagnosis. However, due to limitations in MRI scanning time, cost, hardware, and other factors, the spatial resolution of MRI images often cannot meet the diagnostic needs of doctors. Defects and shortcomings of existing technologies:

[0004] 1. It is usually very expensive to obtain higher resolution medical image information by upgrading hardware, and the use of expensive medical equipment often increases the economic burden.

[0005] 2. For the field of medical imaging, taking medical images with higher spatial resolution also means that patients will be exposed to radiation at a higher dose and for a longer period of time, which undoubtedly adds a considerable burden to the health of patients and radiologists.

[0006] 3. The current image super-resolution methods based on generative adversarial networks often have good visual effects, but their generated results often produce texture information that does not exist in actual high-resolution images. This inaccurate and unrealistic texture information may improve the visual effect in natural images, but in medical images it is likely to cause doctors to misjudge the condition, which is undoubtedly fatal to patients.

[0007] Therefore, there is a need to provide a medical image super-resolution method and system based on feature compensated attention to generate high-resolution medical images that are closer to real images. Summary of the invention

[0008] The present invention provides a medical image super-resolution method based on feature-compensated attention, comprising: establishing a medical image super-resolution generative adversarial network model, wherein the medical image super-resolution generative adversarial network model at least includes a generator and a discriminator, the generator at least includes a first feature extraction component, a feature injection component, a second feature extraction component and a reconstruction component, the outputs of the first feature extraction component and the feature injection component are inputs of the second feature extraction component, and the output of the second feature extraction component is input of the reconstruction component; training the medical image super-resolution generative adversarial network model; obtaining a medical image super-resolution model based on the trained medical image super-resolution generative adversarial network model, wherein the medical image super-resolution model includes the generator of the trained medical image super-resolution generative adversarial network model; acquiring a low-resolution medical image; and obtaining a high-resolution medical image based on the low-resolution medical image through the medical image super-resolution model.

[0009] Furthermore, the first feature extraction component includes a 3×3 convolutional layer.

[0010] Furthermore, the second feature extraction component includes a plurality of feature compensation attention units arranged in sequence, wherein the feature compensation attention unit includes a spatial feature compensation attention layer, a channel feature compensation attention layer and a convolutional layer, the output of the first feature extraction component serves as the input of the first feature compensation attention unit, and the output of the feature injection component serves as the input of each feature compensation attention unit.

[0011] Furthermore, the feature injection unit includes a convolution layer, a nonlinear activation layer, an upsampling layer and a downsampling layer. The feature injection unit is used to extract the compensation features of the low-resolution medical image. The output of the feature injection unit is:

[0012] DR feature =conv(concat(U,V)), U=down(CNN(up(DR in ))), V=conv(relu(conv(relu(relu(conv(DR in )))))+DR in , among which, DR featureis the compensation feature extracted from the low-resolution medical image, conv() is the convolution layer, concat() is the concatenation function, U is the first feature, V is the second feature, down() is the downsampling layer, CNN() is, up() is the upsampling layer, DR in ∈R h×w×c is the input information of the feature injection unit, relu() is the nonlinear activation function, h is the height of the low-resolution medical image, w is the width of the low-resolution medical image, and c is the number of channels of the low-resolution medical image.

[0013] Furthermore, the feature injection unit includes a convolution layer and a nonlinear activation layer, and the feature injection unit is used to extract the compensation feature of the low-resolution medical image to train the medical image super-resolution generative adversarial network model, including: inputting the sample low-resolution medical image into the generator of the medical image super-resolution generative adversarial network model, a first feature extraction component extracting the first feature of the sample low-resolution medical image, the feature injection component extracting the compensation feature from the sample low-resolution medical image, the second feature extraction component extracting the second feature of the sample low-resolution medical image based on the first feature of the sample low-resolution medical image and the compensation feature and performing nonlinear mapping, the reconstruction component generating a high-resolution medical image corresponding to the sample low-resolution medical image based on the output of the second feature extraction component; inputting the generated high-resolution medical image and the real high-resolution medical image corresponding to the sample low-resolution medical image into the discriminator of the medical image super-resolution generative adversarial network model to obtain a discrimination result; calculating the overall loss based on the generated high-resolution medical image, the real high-resolution medical image corresponding to the sample low-resolution medical image and the discrimination result; optimizing the medical image super-resolution generative adversarial network model based on the overall loss until a preset condition is met.

[0014] Furthermore, the loss function is: Loss = AW (GLoss, L1Loss, PerceptualLoss; θ), wherein Loss is the total loss, GLoss is the generator loss, L1Loss is the L1 loss, PerceptualLoss is the perceptual loss, AW() is the automatic weight module, and θ is a learnable tensor including the parameterized contribution coefficient of the generator loss, the contribution coefficient of the L1 loss, and the contribution coefficient of the perceptual loss.

[0015] Furthermore, the L1 loss is: Among them, w is the width of the real high-resolution medical image, h is the height of the real high-resolution medical image, HR is the real high-resolution medical image, and SR is the generated high-resolution medical image.

[0016] Furthermore, the perceptual loss is: Where N is the number of network layers used, λ i is the weight of the i-th layer, φ i (HR) is the feature map generated by the i-th layer of the VGG network with a real high-resolution medical image as input, and φ i (SR) is the feature map produced by the i-th layer of the VGG network with the generated high-resolution medical image as input, It is the L2 norm operation.

[0017] Furthermore, a high-resolution medical image is obtained based on the low-resolution medical image through the medical image super-resolution model, including: dividing the low-resolution medical image into multiple low-resolution medical image slices in the channel dimension; and obtaining a high-resolution medical image based on the multiple low-resolution medical image slices through the medical image super-resolution model.

[0018] The present invention provides a medical image super-resolution system based on feature-compensated attention, comprising: a model building module, used to build a medical image super-resolution generative adversarial network model, wherein the medical image super-resolution generative adversarial network model includes at least a generator and a discriminator, and the generator includes at least a first feature extraction component, a feature injection component, a second feature extraction component and a reconstruction component, the outputs of the first feature extraction component and the feature injection component are the inputs of the second feature extraction component, and the outputs of the second feature extraction component are the inputs of the reconstruction component; a model training module, used to train the medical image super-resolution generative adversarial network model; a model generation module, used to obtain a medical image super-resolution model based on the trained medical image super-resolution generative adversarial network model, wherein the medical image super-resolution model includes the generator of the trained medical image super-resolution generative adversarial network model; an image acquisition module, used to acquire a low-resolution medical image; and an image super-resolution module, used to obtain a high-resolution medical image based on the low-resolution medical image through the medical image super-resolution model.

[0019] Compared with the prior art, the medical image super-resolution method and system based on feature compensation attention provided by the present invention have at least the following beneficial effects:

[0020] The generator of the medical image super-resolution generative adversarial network model can map the input low-resolution image to the corresponding high-resolution image. The discriminator judges whether the input image is a real image and feeds it back to the generator, so that the generator can produce a generation result with a visual effect closer to the real image. Through the synergy of the feature injection component and the second feature extraction component, the generator's generation result is not only closer to the real image in visual effect, but also its feature texture is more faithful to the real high-resolution image. Compared with the existing medical image super-resolution method, the generative adversarial network structure is used to improve the visual effect of the network model to generate pictures. At the same time, a feature compensation attention mechanism is proposed, which can effectively and timely compensate for the feature information lost in the network transmission of image information, so that the final generated high-resolution image has more realistic texture information, which is of great significance for clinical diagnosis and for doctors to make more accurate diagnoses. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0022] Figure 1 is a flowchart of a medical image super-resolution method based on feature compensation attention according to some embodiments of this specification;

[0023] Figure 2 is a schematic diagram of the structure of a medical image super-resolution generative adversarial network model according to some embodiments of this specification;

[0024] Figure 3 is a schematic diagram of the structure of a generator according to some embodiments of this specification;

[0025] Figure 4 is a schematic diagram of the structure of the spatial feature compensation attention layer according to some embodiments of this specification;

[0026] Figure 5 is a schematic diagram of the structure of the channel feature compensation attention layer according to some embodiments of this specification;

[0027] Figure 6 It is a module diagram of a medical image super-resolution system based on feature-compensated attention according to some embodiments of the present specification. DETAILED DESCRIPTION

[0028] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0029] Figure 1 is a flowchart of a medical image super-resolution method and system based on feature compensation attention according to some embodiments of this specification, such as Figure 1 As shown, the medical image super-resolution method based on feature compensated attention may include the following steps.

[0030] Step 110, establishing a medical image super-resolution generative adversarial network model.

[0031] Figure 2 is a schematic diagram of the structure of a medical image super-resolution generative adversarial network model according to some embodiments of this specification, such as Figure 2 As shown, the medical image super-resolution generative adversarial network model includes at least a generator and a discriminator. At the same time, for medical images, the magnetic resonance images used in the present invention, specifically the T1 images in the IXI dataset, are decomposed in the channel dimension to obtain several image slices of the same size for training and experiments.

[0032] Figure 3 is a schematic diagram of the structure of a generator according to some embodiments of this specification, such as Figure 3 As shown, the generator includes at least a first feature extraction component, a feature injection component, a second feature extraction component and a reconstruction component. The outputs of the first feature extraction component and the feature injection component are inputs of the second feature extraction component, and the output of the second feature extraction component is input of the reconstruction component.

[0033] In some embodiments, the first feature extraction component includes a 3×3 convolution layer. The first feature extraction component is composed of a simple 3×3 convolution layer, which can control the size of the medical image super-resolution generative adversarial network model and reduce the amount of calculation. The first feature extraction component is used to extract the first features of the input image, including shallow features such as contours and boundaries, basic texture information, and simple shapes, and prepare for feature extraction and spatial mapping of subsequent modules.

[0034] The output of the first feature extraction component is:

[0035] LR feature =SFEM(LR)

[0036] Among them, LR feature The first feature output by the first feature extraction component, LR feature ∈R h×w×e , SFEM( ) is the first feature extraction component, LR is the input image, LR∈R h×w×c , h is the height of the input image, w is the width of the input image, c is the number of channels of the input image, and e represents the number of channels of the first feature.

[0037] In some embodiments, the second feature extraction component includes a plurality of feature compensation attention units arranged in sequence, and the second feature extraction component is used to extract the second feature of the sample low-resolution medical image based on the first feature and the compensation feature of the input image and perform nonlinear mapping.

[0038] Among them, the feature compensation attention unit includes a spatial feature compensation attention layer, a channel feature compensation attention layer and a convolutional layer. The output of the first feature extraction component is used as the input of the first feature compensation attention unit, and the output of the feature injection component is used as the input of each feature compensation attention unit.

[0039] Figure 4 is a schematic diagram of the structure of the spatial feature compensation attention layer according to some embodiments of this specification, such as Figure 4 As shown in the figure, the spatial feature compensation attention layer first aggregates the information of the channel dimension of the input feature image so that it temporarily contains only the spatial feature information. Then, based on the converted feature information, the values ​​of Q and K are calculated in the spatial dimension, and the compensation information passed by the feature injection component is also aggregated from the information of the channel dimension. Then, V is calculated based on the value, and finally, Q (Query), K (Key), and V (Value) are subjected to spatial attention operation:

[0040]

[0041] Q=X·W Q #

[0042] K=X·W k #

[0043] V=feature i ·W v #

[0044] Among them, Atten Spatial (Q,K,V) is the spatial attention operation, X∈R h×w×1 is the input of the spatial feature compensation attention layer after the channel information of the input feature image is aggregated, W Q , W k and W V is a learnable linear mapping weight, feature i ∈Rh×w×1 is the feature information of the i-th channel in the output of the feature injection component.

[0045] Figure 5 is a schematic diagram of the structure of the channel feature compensation attention layer according to some embodiments of this specification, such as Figure 5 As shown in the figure, unlike the spatial feature compensation attention layer, the channel feature compensation attention layer first aggregates the input feature information in the spatial dimension, so that it temporarily only contains information in the channel dimension, and calculates the values ​​of Q and K based on this, while the value of V is still provided by the feature injection component.

[0046]

[0047] Among them, Atten Channel (Q, K, V) is the channel attention operation.

[0048] In some embodiments, the feature injection unit includes a convolution layer, a nonlinear activation layer, an upsampling layer and a downsampling layer, and the feature injection unit is used to extract compensation features of the low-resolution medical image.

[0049] The output of the feature injection unit is:

[0050] DR feature =conv(concat(U,V))

[0051] U=down(CNN(up(DR in )))

[0052] V=conv(relu(conv(relu(conv(DR in )))))+DR in

[0053] Among them, DR feature is the compensation feature extracted from the low-resolution medical image, conv() is the convolution layer, concat() is the concatenation function, U is the first feature, V is the second feature, down() is the downsampling layer, CNN() is, up() is the upsampling layer, DR in ∈R h×w×c is the input information of the feature injection unit, relu() is the nonlinear activation function, h is the height of the low-resolution medical image, w is the width of the low-resolution medical image, and c is the number of channels of the low-resolution medical image.

[0054] Specifically, the feature injection unit is a trapezoidal structure branch, which is mainly composed of a convolutional neural network. According to the convolution theorem, the convolution of two signals in the spatial domain is equivalent to their product in the frequency domain, which can be expressed as follows:

[0055]

[0056] Among them, x(t) and h(t) are two spatial domain signals, y(t) is their convolution result, and τ is a dummy variable in the integration, which is used to represent the time or spatial position in the integration process.

[0057] According to the convolution theorem, this equation can be equivalent to the following equation:

[0058] Y(f)=X(f)H(f)

[0059] Among them, Y(f) is the expression of y(t) in the frequency domain, X(f) and H(f) are the expressions of x(t) and h(t) in the frequency domain respectively. At the same time, due to the overall small size of the convolution kernel, the convolutional neural network can extract the local high-frequency part of the image information, and the attention mechanism is more inclined to the global low-frequency information according to existing research findings. Therefore, the feature injection module can form a good complementary effect with the backbone of the network.

[0060] In the feature injection unit, the input features are divided into two parts by channel average splitting, and the upper part is directly sent to the convolutional neural network. In order to obtain more detailed high-frequency information, consider extracting information from feature information with higher dimensions and resolutions, so the lower part is upsampled to obtain higher-resolution feature information and sent to the convolutional neural network. Finally, the information is downsampled back to the original resolution, and the two parts are spliced ​​to obtain the final extracted high-frequency information. In summary, the feature injection unit can extract feature information that is lost during the propagation of low-resolution images in the backbone network.

[0061] The reconstruction component consists of bicubic interpolation and convolution layers, and is used to generate a high-resolution medical image corresponding to the sample low-resolution medical image based on the output of the second feature extraction component.

[0062] Step 120: training a medical image super-resolution generative adversarial network model.

[0063] Specifically include:

[0064] The sample low-resolution medical image is input into the generator of the medical image super-resolution generative adversarial network model, the first feature extraction component extracts the first feature of the sample low-resolution medical image, the feature injection component extracts the compensation feature from the sample low-resolution medical image, the second feature extraction component extracts the second feature of the sample low-resolution medical image based on the first feature and the compensation feature of the sample low-resolution medical image and performs nonlinear mapping, and the reconstruction component generates a high-resolution medical image corresponding to the sample low-resolution medical image based on the output of the second feature extraction component;

[0065] Inputting the generated high-resolution medical image and the real high-resolution medical image corresponding to the sample low-resolution medical image into the discriminator of the medical image super-resolution generative adversarial network model to obtain a discrimination result;

[0066] Calculate the overall loss based on the generated high-resolution medical image, the real high-resolution medical image corresponding to the sample low-resolution medical image, and the discrimination result;

[0067] The medical image super-resolution generative adversarial network model is optimized based on the overall loss, and the generator is trained using gradient backpropagation until the preset conditions are met, such as loss convergence.

[0068] Specifically, the three-dimensional sample low-resolution medical image is sliced ​​into multiple sample low-resolution medical image slices. The multiple sample low-resolution medical image slices are used as inputs of the first feature extraction component. When the magnification factor is 4, the size of the input low-resolution medical image slice is 64×64, and when the magnification factor is 2, the size of the input low-resolution medical image slice is 128×128.

[0069] In some embodiments, the loss function is:

[0070] Loss=GLoss+L1Loss+PerceptualLoss

[0071] In practice, it is found that different losses contribute differently to model training, so the losses cannot be simply added together to get the total loss. Instead, a coefficient should be assigned to each loss to balance the contribution between different losses:

[0072] Loss=λ1GLoss+λ2L1Loss+λ3PerceptualLoss

[0073] Among them, λ1, λ2 and λ3 are the contribution coefficients of the generator loss, L1 loss and perceptual loss respectively.

[0074] However, it is difficult to adjust the contribution coefficient manually. It is difficult to adjust to a suitable value without prior experience. Therefore, the contribution coefficient is parameterized to make the loss function a part of the neural network. The Auto-Weight Module is introduced to adapt the value of the contribution coefficient. The parameterization process can be equivalent to the following equation:

[0075] Loss=AW(GLoss,L1Loss,PerceptualLoss;θ)

[0076] Among them, Loss is the total loss, GLoss is the generator loss, L1Loss is the L1 loss, PerceptualLoss is the perceptual loss, AW( ) is the automatic weight module, and θ is a learnable tensor, including the contribution coefficient of the parameterized generator loss, the contribution coefficient of the L1 loss, and the contribution coefficient of the perceptual loss.

[0077] In some embodiments, the generator loss is:

[0078] GLoss = -log(D(G(LR)))

[0079] DLoss=-logD(HR)-log(1-D(G(LR)))

[0080] Among them, D( ) is the operator symbol of the discriminator, G( ) is the operator symbol of the generator, and LR is the low-resolution medical image corresponding to the real high-resolution medical image.

[0081] In some embodiments, the L1 loss is:

[0082]

[0083] Among them, w is the width of the real high-resolution medical image, h is the height of the real high-resolution medical image, HR is the real high-resolution medical image, and SR is the generated high-resolution medical image.

[0084] In some embodiments, the perceived loss is:

[0085]

[0086] Where N is the number of network layers used, λ i is the weight of the i-th layer, φ i (HR) is the feature map generated by the i-th layer of the VGG (Visual Geometry Group) network with a real high-resolution medical image as input, and φ i (SR) is the feature map produced by the i-th layer of the VGG network with the generated high-resolution medical image as input, It is the L2 norm operation.

[0087] Step 130, obtaining a medical image super-resolution model based on the trained medical image super-resolution generative adversarial network model.

[0088] Among them, the medical image super-resolution model includes a generator of a trained medical image super-resolution generative adversarial network model.

[0089] Step 140: Acquire a low-resolution medical image.

[0090] Among them, the low-resolution medical image may be an MRI image, a CT image, or the like.

[0091] Step 150: Obtain a high-resolution medical image based on the low-resolution medical image using a medical image super-resolution model.

[0092] Specifically include:

[0093] Dividing the low-resolution medical image into a plurality of low-resolution medical image slices in the channel dimension;

[0094] A high-resolution medical image is obtained based on multiple low-resolution medical image slices through a medical image super-resolution model.

[0095] The beneficial effects of this method are explained below in conjunction with experiments.

[0096] The experiment was conducted using a public dataset that contains more than 580 MR images from normal healthy subjects. By comparing a variety of natural image super-resolution methods and medical image super-resolution methods, the objective evaluation indicators and subjective visual effects were compared. The objective evaluation indicators used were peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM).

[0097] The peak signal-to-noise ratio indicates the ratio of the maximum possible signal power to the destructive noise power that affects its representation accuracy. Generally, the larger the peak signal-to-noise ratio, the closer the two images are. The formula is:

[0098]

[0099] Among them, MAX represents the maximum value of the image point, such as in natural images, MAX is equal to 255. MSE represents the mean square error, and its calculation method is given by the following formula:

[0100]

[0101] Among them, m and n represent the length and width of the image respectively, I represents the real image, and K represents the noisy image or the image generated by the neural network.

[0102] The structural similarity index is used to measure the similarity between two images. Generally, the higher the better. SSIM=1 means that the two images are completely identical. The formula is:

[0103]

[0104] Among them, x and y represent two images from different sources, μ x is the expectation of x, μy is the expectation of y, are the variances of x and y, σ xy represents the covariance of x and y, c1 and c2 are constants used to maintain stability.

[0105] This method is compared with other methods in terms of medical image super-resolution effects under the dataset, and the comparison is made at magnification factors of 2 and 4. The bicubic interpolation method is used to obtain the corresponding low-resolution images from the high-resolution real medical images, and the low-resolution images are used as the input of the model to the medical image super-resolution models of all selected comparison methods and this method. The comparison methods used include: Super-Resolution Convolutional Neural Network (SRCNN), Enhanced Super-Resolution Generative Adversarial Network (Real-ESRGAN) and Enhanced Deep Super-Resolution with Multi-level Multi-Head Cross-Attention (EDSR+MMHCA), and the bicubic interpolation method (Bicubic) is used as a benchmark for comparison. They are mainly based on CNN and Transformer design. Among them, Real-ESRGAN has extremely good visual effects and is very competitive in the field of natural image super-resolution, while the EDSR+MMHCA method is one of the most advanced methods in the field of medical image super-resolution. All methods are trained and tested on the same dataset, and all experimental results are objective and fair. The experimental results are shown in Tables 1 and 2.

[0106] Table 1

[0107]

[0108] Table 2

[0109]

[0110]

[0111] According to the experimental results in Table 1 and Table 2, this method can obtain better results in both 4x and 2x magnification, and achieves the best results in both PSNR and SSIM indicators. It is particularly important to note that Real-ESRGAN uses the structure of a generative adversarial network, which leads to low quantitative indicators. This is a defect of the generative adversarial network. Although this method is still based on the structure of a generative adversarial network, its quantitative indicators are higher than Real-ESRGAN, indicating that the high-resolution medical images generated by this method will be closer to real high-resolution medical images.

[0112] The high-resolution medical images generated by this method are clearer and have better visual effects than those of SRCNN and EDSR+MMHCA, the most advanced method in the field of medical image super-resolution. Compared with Real-ESRGAN, the contrast of texture information is indeed insufficient, but the texture of the output result of Real-ESRGAN is quite different from the real image. In the field of natural image super-resolution, this difference may provide a good visual effect, but in the field of medical image super-resolution, even this slight difference may be fatal. The high-resolution medical images generated by this method are closer to the real high-resolution images in terms of texture details, and the output results are faithful to the real images, which can assist doctors in making correct diagnoses.

[0113] Figure 6 is a module schematic diagram of a medical image super-resolution system based on feature compensation attention according to some embodiments of this specification, such as Figure 6 As shown, the medical image super-resolution system based on feature-compensated attention may include a model building module, a model training module, a model generation module, an image acquisition module and an image super-resolution module.

[0114] The model building module can be used to establish a medical image super-resolution generative adversarial network model, wherein the medical image super-resolution generative adversarial network model includes at least a generator and a discriminator, the generator includes at least a first feature extraction component, a feature injection component, a second feature extraction component and a reconstruction component, the output of the first feature extraction component and the feature injection component is the input of the second feature extraction component, and the output of the second feature extraction component is the input of the reconstruction component;

[0115] The model training module can be used to train the medical image super-resolution generative adversarial network model;

[0116] The model generation module can be used to obtain a medical image super-resolution model based on the trained medical image super-resolution generative adversarial network model, wherein the medical image super-resolution model includes a generator of the trained medical image super-resolution generative adversarial network model;

[0117] The image acquisition module can be used to acquire low-resolution medical images;

[0118] The image super-resolution module can be used to obtain high-resolution medical images based on low-resolution medical images through a medical image super-resolution model.

[0119] The medical image super-resolution system based on feature compensated attention can be used to execute the medical image super-resolution method based on feature compensated attention. For more description of the medical image super-resolution system based on feature compensated attention, please refer to the relevant description of the medical image super-resolution method based on feature compensated attention, which will not be repeated here.

[0120] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A medical image super-resolution method based on feature-compensated attention, characterized in that: include: Establishing a medical image super-resolution generative adversarial network model, wherein the medical image super-resolution generative adversarial network model at least includes a generator and a discriminator, the generator at least includes a first feature extraction component, a feature injection component, a second feature extraction component and a reconstruction component, the outputs of the first feature extraction component and the feature injection component are the inputs of the second feature extraction component, and the output of the second feature extraction component is the input of the reconstruction component; Training the medical image super-resolution generative adversarial network model; Based on the trained medical image super-resolution generative adversarial network model, a medical image super-resolution model is obtained, wherein the medical image super-resolution model includes a generator of the trained medical image super-resolution generative adversarial network model; Acquisition of low-resolution medical images; A high-resolution medical image is obtained based on the low-resolution medical image through the medical image super-resolution model.

2. The medical image super-resolution method based on feature compensation attention according to claim 1, characterized in that: The first feature extraction component includes a 3×3 convolutional layer.

3. The medical image super-resolution method based on feature compensation attention according to claim 1, characterized in that: The second feature extraction component includes a plurality of feature compensation attention units arranged in sequence, wherein the feature compensation attention unit includes a spatial feature compensation attention layer, a channel feature compensation attention layer and a convolutional layer, the output of the first feature extraction component serves as the input of the first feature compensation attention unit, and the output of the feature injection component serves as the input of each feature compensation attention unit.

4. The medical image super-resolution method based on feature compensation attention according to claim 1, characterized in that: The feature injection unit includes a convolution layer, a nonlinear activation layer, an upsampling layer and a downsampling layer. The feature injection unit is used to extract the compensation features of the low-resolution medical image. The output of the feature injection unit is: DR feature =conv(concat(U,V)) U=down(CNN(up(DR in ))) V=conv(relu(conv(relu(conv(DR in )))))+DR in Among them, DR feature is the compensation feature extracted from the low-resolution medical image, conv() is the convolution layer, concat() is the concatenation function, U is the first feature, V is the second feature, down() is the downsampling layer, CNN() is, up() is the upsampling layer, DR in ∈R h×w×c is the input information of the feature injection unit, relu() is the nonlinear activation function, h is the height of the low-resolution medical image, w is the width of the low-resolution medical image, and c is the number of channels of the low-resolution medical image.

5. The medical image super-resolution method based on feature-compensated attention according to any one of claims 1 to 4, characterized in that: Training the medical image super-resolution generative adversarial network model comprises: The sample low-resolution medical image is input into the generator of the medical image super-resolution generative adversarial network model, the first feature extraction component extracts the first feature of the sample low-resolution medical image, the feature injection component extracts the compensation feature from the sample low-resolution medical image, the second feature extraction component extracts the second feature of the sample low-resolution medical image based on the first feature of the sample low-resolution medical image and the compensation feature and performs nonlinear mapping, and the reconstruction component generates a high-resolution medical image corresponding to the sample low-resolution medical image based on the output of the second feature extraction component; Inputting the generated high-resolution medical image and the real high-resolution medical image corresponding to the sample low-resolution medical image into the discriminator of the medical image super-resolution generative adversarial network model to obtain a discrimination result; Calculating the overall loss based on the generated high-resolution medical image, the real high-resolution medical image corresponding to the sample low-resolution medical image, and the discrimination result; The medical image super-resolution generative adversarial network model is optimized based on the overall loss until a preset condition is met.

6. The medical image super-resolution method based on feature compensation attention according to claim 5, characterized in that: The loss function is: Loss=AW(GLoss,L1Loss,PerceptualLoss;θ) Among them, Loss is the total loss, GLoss is the generator loss, L1Loss is the L1 loss, PerceptualLoss is the perceptual loss, AW() is the automatic weight module, and θ is a learnable tensor, including the contribution coefficient of the parameterized generator loss, the contribution coefficient of the L1 loss, and the contribution coefficient of the perceptual loss.

7. The medical image super-resolution method based on feature compensation attention according to claim 6, characterized in that: The L1 loss is: Among them, w is the width of the real high-resolution medical image, h is the height of the real high-resolution medical image, HR is the real high-resolution medical image, and SR is the generated high-resolution medical image.

8. The medical image super-resolution method based on feature-compensated attention according to claim 7, characterized in that: The perceptual loss is: Where N is the number of network layers used, λ i is the weight of the i-th layer, φ i (HR) is the feature map generated by the i-th layer of the VGG network with a real high-resolution medical image as input, and φ i (SR) is the feature map produced by the i-th layer of the VGG network with the generated high-resolution medical image as input, It is the L2 norm operation.

9. The medical image super-resolution method based on feature-compensated attention according to any one of claims 1 to 4, characterized in that: Obtaining a high-resolution medical image based on the low-resolution medical image by using the medical image super-resolution model, including: Dividing the low-resolution medical image into a plurality of low-resolution medical image slices in a channel dimension; A high-resolution medical image is obtained based on the multiple low-resolution medical image slices through the medical image super-resolution model.

10. A medical image super-resolution system based on feature-compensated attention, characterized in that: include: A model building module, used to build a medical image super-resolution generative adversarial network model, wherein the medical image super-resolution generative adversarial network model at least includes a generator and a discriminator, the generator at least includes a first feature extraction component, a feature injection component, a second feature extraction component and a reconstruction component, the outputs of the first feature extraction component and the feature injection component are the inputs of the second feature extraction component, and the output of the second feature extraction component is the input of the reconstruction component; A model training module, used for training the medical image super-resolution generative adversarial network model; A model generation module, used to obtain a medical image super-resolution model based on the trained medical image super-resolution generative adversarial network model, wherein the medical image super-resolution model includes a generator of the trained medical image super-resolution generative adversarial network model; An image acquisition module, used for acquiring low-resolution medical images; The image super-resolution module is used to obtain a high-resolution medical image based on the low-resolution medical image through the medical image super-resolution model.

Images