A medical image enhancement method based on a multi-scale attention generative adversarial network

Abstract

The application discloses a medical image enhancement method based on a multi-scale attention generative adversarial network, two generators and two discriminators are trained in the confrontation, attention gates are added when encoding layer skip connection to filter invalid information, highlight important image areas, and avoid redundant bottom feature accumulation; a pyramid feature FPN is designed to expand the decoding layer, fuse features of different levels and end output; finally, a convolution attention module is used to perform attention allocation on the fused features in the channel and spatial dimensions, and more attention is paid to key information during imaging. The method adds a uniform illumination term, a texture fidelity term, a perception loss term and image smoothness, and strengthens the constraints on the generated image quality from multiple angles such as illumination distribution, texture details, deep semantic features and smoothness to improve the enhancement effect. The application trains the network by using the Adam algorithm, and experiment results verify the feasibility and effectiveness of the application.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of medical image enhancement, medical image synthesis, and belongs to the technical field of medical image processing, and particularly relates to a medical image enhancement method based on a multi-scale attention generative adversarial network. BACKGROUND

[0002] With the continuous improvement of modern medical level, medical imaging equipment as the core of "intelligent medical treatment" is gradually applied in various primary hospitals. X-ray, computer tomography (CT), magnetic resonance (MR), confocal laser scanning microscope (CLSM), ultrasound and other medical images can often provide doctors with a lot of information related to biological or anatomical tissues, and are important basis for clinical diagnosis and treatment. However, there are many uncontrollable factors in the process of actually obtaining medical images, resulting in low-quality images collected, the main reasons including: (1) due to cost and technology constraints, the imaging equipment is out of repair, lacks maintenance, and may not have the latest functional components; (2) the environment of the imaging equipment is poor, such as darkness, underexposure, and existence of shielding. These low-quality images have two adverse effects: (1) greatly increasing the energy and time cost of medical staff reading and identifying images, thereby leading to uncertainty, misjudgment and omission rate of clinical diagnosis; (2) the accuracy performance of computer vision model is greatly reduced when facing low-quality images, which is not conducive to subsequent image analysis, segmentation and classification tasks, and greatly reduces the application value of medical images. Therefore, it is very important to ensure the high quality of medical images, and it has become a research hotspot in recent years to improve the quality of medical images by using the rapidly developing computer vision technology. The main purpose of medical image enhancement is to uniform image brightness, ensure image structure, restore image texture details, and more clearly display image content, so that it becomes a high-quality image more suitable for naked eye observation and computer vision model use, which has important significance for medical and computer vision fields.

[0003] The problem of medical image enhancement has attracted widespread academic attention, and the current traditional medical image enhancement method has obvious defects. When processing the region, the foreground and background are usually not distinguished, resulting in over-enhancement of high-quality regions, and the low-quality regions are not obviously enhanced, and even the noise is enlarged, and the detail retention is poor. Most of the medical image enhancement methods based on deep learning are completely supervised, which need to be trained based on strictly paired image data; while the unpaired method has stronger applicability, but still has challenges in maintaining texture details and balancing background, so it needs to be redesigned. SUMMARY

[0004] (I) Technical problems solved

[0005] In view of the deficiencies of the prior art, the present application provides a medical image enhancement method based on a multi-scale attention generative adversarial network, which can better extract multi-scale and contextual information of medical images, suppress irrelevant activation and highlight key information based on an attention mechanism, and finally optimize constraints on the quality of generated images from multiple angles such as cycle consistency, illumination distribution, texture details, and deep semantics, etc., to solve the problems of relying on strict paired training data and the enhanced images having incomplete structure, uneven illumination distribution, and lost texture details.

[0006] (II) Technical solutions

[0007] To achieve the above-mentioned purpose of better extracting multi-scale and contextual information of medical images, suppressing irrelevant activation and highlighting key information based on an attention mechanism, and finally optimizing constraints on the quality of generated images from multiple angles such as cycle consistency, illumination distribution, texture details, and deep semantics, etc., the present application provides the following technical solutions:

[0008] A medical image enhancement method based on a multi-scale attention generative adversarial network, comprising the following steps:

[0009] S1, constructing a generator G L→H and a discriminator G L→H for an enhanced image branch; constructing the same generator G H→L and discriminator G H→L for a restored image branch;

[0010] S2, inputting an original low-quality image x into the generator G L→H to obtain an enhanced image then inputting into the generator G H→L to obtain a restored low-quality image inputting an original high-quality image y into the generator G H→L to obtain a low-quality image then inputting into the generator G L→H to obtain a restored high-quality image calculating cycle consistency loss L Cyc , illumination uniformity loss L IU , texture fidelity loss L TF , image smoothness loss L IS , and perceptual loss L PL from them;

[0011] S2.1, inputting the original low-quality image x and the original high-quality image y into the generator G H→Land G L→H get the mapping image and identity mapping loss L Idt from which

[0012] S2.2, input the original high-quality image y and the enhanced image to the discriminator D H , and D H discriminate their authenticity, input the original low-quality image x and the generated low-quality image to the discriminator D L , and D L discriminate their authenticity, calculate the adversarial loss L Adv from which

[0013] S3, based on the Adam algorithm and the error back propagation principle, obtain the gradient by optimizing the overall loss function, first fix the generator, update the parameters of the discriminator; then fix the discriminator, update the parameters of the generator; repeat the above process until the upper limit of the training round is reached.

[0014] Preferably, in step S1, the generator takes U-Net based on the convolution residual module as the main body, adds attention gates when jumping connections in the encoding layer to filter invalid information, highlight important image regions, and avoid redundant bottom feature accumulation; design the pyramid feature FPN to expand the decoding layer, fuse features at different levels and the end output, and better utilize multi-scale, context global information; finally, use the convolution attention module to perform attention allocation on the fused features in the channel and spatial dimensions, and pay more attention to key information during imaging.

[0015] Preferably, in step S2, the discriminator is a multi-scale neural network composed of multiple convolution modules.

[0016] Preferably, in step S2, the cycle consistency loss L Cyc , the uniform illumination loss L IU , the texture fidelity loss L TF , the image smoothness loss L IS and the perceptual loss L PL are calculated, including:

[0017] L Cyc = E x [||G H→L (G L→H (x))-x||1]+E y [||G L→H (G H→L (y))-y||1] (1)

[0018]

[0019]

[0020] Preferably, the (1) formula, wherein, x, y respectively represent the original low-quality image and the original high-quality image; G L→H represents the generator for generating high-quality images, G H→L represents the generator for generating low-quality images;

[0021] (2) formula, wherein, E global [·]、 respectively represent the global average illumination of the input image and the local region average illumination of p×p size; represent bicubic interpolation up-sampling operation to eliminate the scale difference;

[0022] (3) formula, wherein, K represents the number of local regions divided by each image; y i , x i respectively represent the i-th local region of the high-quality image y and the low-quality image x, G L→H (x) i , G H→L (y) i respectively represent the i-th local region of the two generated images; and represent the covariance matrix between the i-th region of the original image and the i-th region of the corresponding generated image, respectively y i , G H→L (y) i , x i , G L→H (x) i corresponding standard deviation matrix; c is a small constant for avoiding numerical instability;

[0023] (4) formula, wherein, represents the feature map extracted by the j-th layer convolution after the i-th layer pooling of the pre-trained network; H i,j and W i,j are the size of the feature map;

[0024] (5) formula, wherein, f(x, y) is the intensity function of the given image, H and W are the size of the image described by the intensity function.

[0025] Preferably, in the step S2.2, the adversarial loss L Adv is calculated:

[0026] L Adv = L GAN (G L→H , D Hx, y) + L GAN (G H→L , D L , x, y) (7)

[0027] L GAN (G L→H , D H , x, y) = E x [1 - (D H (G L→H (x))) 2 ] + E y [(D H (y) - 1) 2 ] (8)

[0028] L GAN (G H→L , D L , x, y) = E y [1 - (D L (G H→L (y))) 2 ] + E x [(D L (x) - 1) 2 ] (9).

[0029] Preferably, in the formula (7), (8) and (9), x and y respectively represent the original low-quality image and the original high-quality image; G L→H represents a generator for generating a high-quality image, G H→L represents a generator for generating a low-quality image; D H represents a discriminator for discriminating whether an input image is a high-quality image, D L represents a discriminator for discriminating whether an input image is a low-quality image.

[0030] Preferably, in the step S3, based on the Adam algorithm and the error back propagation principle, the gradient is obtained by optimizing the overall loss function, first fixing the generator, updating the parameters of the discriminator; then fixing the discriminator, updating the parameters of the generator; repeating the above process until the upper limit of the training round is reached:

[0031] Preferably, in the formula (10) and (11), λ1, λ2, α, β, γ and η respectively represent the weights corresponding to the cycle-consistency loss L Cyv , the identity mapping loss L Idt , the uniform illumination loss L IU , the texture fidelity loss L TF , the perceptual loss L PL and the image smoothing loss L IS .

[0032] (III) Beneficial effects

[0033] Compared with the prior art, the medical image enhancement method based on the multi-scale attention generative adversarial network provided by the application has the following beneficial effects:

[0034] 1. The medical image enhancement method based on the multi-scale attention generative adversarial network solves the problem of non-paired medical image enhancement by introducing the multi-scale attention-based generative adversarial network, can better extract multi-scale and context information of the medical image, suppresses irrelevant activation and highlights key information based on the attention mechanism, and finally optimizes the constraints on the quality of the generated image from multiple angles such as cycle consistency, illumination distribution, texture details and deep semantics.

[0035] 2. The medical image enhancement method based on the multi-scale attention generative adversarial network has excellent enhancement effect, can obtain higher-quality medical images, is beneficial to doctors for clinical diagnosis and treatment and subsequent analysis, and has higher application value. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is the overall process and principle schematic structure of the medical image enhancement method of the embodiment of the application;

[0037] Figure 2 is a network structure diagram of the generator of the medical image enhancement method of the embodiment of the application;

[0038] Figure 3 is a convolution residual module structure diagram in the generator of the medical image enhancement method of the embodiment of the application;

[0039] Figure 4 is an up-sampling module structure diagram in the generator of the medical image enhancement method of the embodiment of the application;

[0040] Figure 5 is an attention gate AG structure diagram in the generator of the medical image enhancement method of the embodiment of the application;

[0041] Figure 6 is a pyramid feature FPN structure diagram in the generator of the medical image enhancement method of the embodiment of the application;

[0042] Figure 7 is a convolution attention module CBAM structure diagram in the generator of the medical image enhancement method of the embodiment of the application;

[0043] Figure 8 is a network structure diagram of the discriminator of the medical image enhancement method of the embodiment of the application;

[0044] Figure 9is a comparison chart of the light distribution of the enhanced image of the embodiment of the present application and the enhanced image of other advanced methods;

[0045] Figure 10 is a comparison chart of the texture details of the enhanced image of the embodiment of the present application and the enhanced image of other advanced methods;

[0046] Figure 11 is a comparison chart of the residual image noise of the enhanced image of the embodiment of the present application and the enhanced image of other advanced methods;

[0047] Figure 12 is a comparison chart of the segmentation effect of the enhanced image of the embodiment of the present application and the enhanced image of other advanced methods;

[0048] Figure 13 is a schematic diagram of the segmentation result of the medical image enhancement method of the embodiment. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0050] Embodiment:

[0051] The medical image enhancement method based on the multi-scale attention generative adversarial network provided by the embodiment of the present application is suitable for non-paired medical image enhancement, as shown in Figure 1 , which comprises the following steps:

[0052] Step 1, based on the idea framework of non-paired medical image enhancement, a generator G L→H and a discriminator D H are constructed for the enhanced image branch; a same generator G H→L and a discriminator D L are constructed for the restored image branch.

[0053] Among them, the generator structure is shown in Figure 2 , which is a U-Net type encoder-decoder structure as a whole. Specifically, each encoding layer is composed of a convolution residual module followed by a maximum pooling layer, and each decoding layer is composed of a bilinear interpolation up-sampling module followed by a convolution residual module. The convolution residual module used by the encoder and the decoder has the same structure, as shown in Figure 3As shown, containing two stacked 3x3 convolution-batch normalization BN-LeakyReLU, using a short distance connection (Shortcut Connection) between the input value and the output value to realize the residual structure, which can avoid the problems of network degradation, gradient disappearance and explosion caused by the increase of depth. The decoder upsampling module is shown in Figure 4 As shown, it is a series structure of double upsampling-3x3 convolution-batch normalization BN-LeakyReLU-Dropout.

[0054] As shown in Figure 2 As shown, for the same level of encoding layer and decoding layer data, use the jump connection to transfer information, effectively use the spatial information between the encoding and decoding paths. And build an attention gate (Attention Gate, AG) at each decoding layer, AG will filter the features transmitted from the encoding layer to the decoding layer, apply attention mechanism to retain important information and participate in subsequent decoding calculation. Compared with direct simple addition, AG can suppress irrelevant activation response and highlight important image areas, effectively avoiding the accumulation of redundant low-level features. Its structure is shown in Figure 5 As shown.

[0055] In the decoding upsampling path of the traditional U-Net, each decoding layer outputs high-level semantic features of different scales, but it ignores the influence of different scale feature maps and only uses the output of the last layer as the generation result. In addition, the features between different scale levels cannot be shared. In order to solve this problem, this paper designs a feature pyramid (Feature Pyramid Network, FPN) to extend the U-Net, adds an additional path composed of 1x1 convolution layer and upsampling layer at each level of the decoder. The purpose is to extract multi-scale features from the 7 levels of the decoder and eliminate the scale difference, then stack in the channel dimension and input into the 3x3 convolution to get the features fused with multi-scale information, and finally element-wise add with the output of the last layer. As shown in Figure 6 As shown, Figure 5 The FPN design of the present application is more clearly presented. Compared with the traditional FPN which gradually fuses high-level features from the bottom layer by 1x1 convolution-upsampling, the FPN of the present application is less likely to lose position information in the fusion process of bottom-level features and deep-level semantics, and better utilizes multi-scale and context global information.

[0056] Before obtaining the generated image, a lightweight convolutional attention module (Convolutional Block Attention Module, CBAM) is added to the output features of the decoder for attention calculation in the channel and spatial position dimensions. The CBAM structure is shown in Figure 7As shown, it is a series structure of a channel attention module (CAM) and a spatial attention module (SAM) as a whole. The design ideas of the channel attention module CAM and the spatial attention module SAM are basically the same: first, try to use average and maximum pooling to compress and filter invalid information, then generate an attention map through a convolution layer and a Sigmoid activation, and finally multiply the original feature to realize adaptive learning of the feature. Only the CAM pays more attention to what kind of feature is more important, and the goal is to complete the allocation of channel attention weights, so global average and maximum pooling are performed based on the spatial dimension, i.e. height and width, to obtain a Cx1x1 channel description. The SAM pays more attention to which position of the feature is more important, and the goal is to complete the allocation of spatial position attention weights, so global average and maximum pooling are performed based on the channel dimension to obtain a 1xHxW spatial description. The use of CBAM can be regarded as a priori guidance for generating images, and more attention can be paid to key information during imaging.

[0057] The discriminator structure is shown in Figure 8 As shown, it is designed in a multi-scale form, and 2x2 maximum pooling, 4x4 maximum pooling and 8x8 maximum pooling are used to extract feature maps of different scales, respectively. A upsampling module is used to eliminate the scale difference, and finally a convolution module is used to fuse multi-scale information.

[0058] Step 2, input the original low-quality image x into the generator G L→H to obtain an enhanced image Then input into the generator G H→L to obtain a restored low-quality image Input the original high-quality image y into the generator G H→L to obtain a low-quality image Then input into the generator G L→H to obtain a restored high-quality image From which the cycle consistency loss L Cyc , the uniform illumination loss L IU , the texture fidelity loss L TF , the image smoothness loss L IS and the perceptual loss L PL are calculated.

[0059] Unlike paired training, unpaired training cannot maintain the output to be the same as the target distribution only with the adversarial loss, and the cycle consistency loss can solve this problem, which requires the enhanced image G L→H (x) to be input into the low-quality generator G H→L to obtain the reconstructed image G H→L (GL→H (x)) as similar as possible to the original low-quality image x, i.e., requiring G H→L (G L→H (x)) ≈ x, G L→H (G H→L (y)) ≈ y. The overall cycle loss is represented as:

[0060] L Cyc = E x [||G H→L (G L→H (x))-x||1]+E y [||G L→H (G H→L (y))-y||1] (1)

[0061] where x, y represent the original low-quality image and the original high-quality image, respectively; G L→H denotes the generator for generating high-quality images, G H→L denotes the generator for generating low-quality images.

[0062] The illumination distribution of the medical image as a whole is very important, and uniform illumination is conducive to the accurate clinical diagnosis and subsequent analysis and treatment of doctors. Therefore, the present application introduces an illumination uniformity term loss function L IU to regularize the illumination distribution of an image, which is achieved by minimizing the illumination difference between local regions and the overall image, and can be regarded as the prior knowledge of human vision. The mathematical expression of the illumination uniformity term is:

[0063]

[0064] where E global [·] and E [·] represent the global average illumination and the p x p size local region average illumination of the input image, respectively; represents a bicubic interpolation up-sampling operation to eliminate the scale difference.

[0065] It is worth mentioning that only the high-quality image needs to satisfy the constraint of illumination regularization, so in the specific training process, the illumination uniformity loss is only calculated in the generator branch of the low-quality image enhancement to high-quality image.

[0066] Although the illumination uniformity term can improve the illumination distribution of the image and make it more suitable for human eye observation, it may lead to too low image contrast, and some key foreground texture details may be homogenized into the background and lose important information. The present application constructs a texture fidelity term loss function L TFNot only the image brightness, contrast and structure are considered, but also the image is divided into multiple regions for local and detailed processing, which can better enhance the texture details of the image, and the mathematical definition is:

[0067]

[0068] Wherein, K represents the number of local regions divided by each image; y i , x i respectively represent the i-th local region of the high-quality image y and the low-quality image x, x L→H (x) i , G H→L (y) i respectively represent the i-th local region of the two generated images; and represent the covariance matrix between the i-th region of the original image and the i-th region of the corresponding generated image, respectively y i , G H→L (y) i , x i , G L→H (x) i corresponding standard deviation matrix; c is a small constant for avoiding numerical instability.

[0069] The cycle consistency constrains the overall structure of the generated image and the real image pixel by pixel at the image level, which is one of the important reasons for the success of GAN. The present application proposes a perception loss term, which attempts to further constrain at the deep feature level of the image. In order to better adapt to the unpaired training task, the specific method is to input the output image and the original image into the pre-trained VGG-16 network model to extract features, and minimize the feature space distance between them, so as to limit the content of the generated image from shifting while ensuring that the generated image has the structure and detail information of the real image. The perception loss is represented as:

[0070]

[0071] Wherein, represents the feature map extracted by the j-th layer convolution after the i-th layer pooling of the pre-trained network; H i,j and W i,j are the size of the feature map.

[0072] The enhanced image may have noise, boundary artifacts and the like, see the red box shown in Figure 9 These are unnecessary interference information, which seriously affects the clinical diagnosis of medical staff. In order to suppress and remove these interferences, this paper introduces an image smoothing term L ISto regularize the network to converge at the right position and ensure the smoothness of the generated image. IS Reference to the total variation principle based on image sharpness prior: the total variation of the image with noise is larger than that without noise. Therefore, for image denoising, reducing the total variation can be achieved, and the image smoothing term is defined as:

[0073]

[0074] where f(x, y) is the intensity function of the given image, H and W are the size of the image described by the intensity function.

[0075] It is worth mentioning that the image described by f(x, y) here is the enhanced high-quality image G L→H (x), that is, the image smoothing loss is only calculated in the generator branch that enhances the low-quality image to high-quality, because there are many unpredictable factors in the process of restoring the high-quality image to the low-quality image, we do not require the low-quality image to also have smoothness. Note that in the actual training process, the weight of the image smoothing term should be set to a small value, if the weight is too high, some texture details will be lost in the smoothing process.

[0076] Step 3, input the original low-quality image x and the original high-quality image y into the generator G H→L and G L→H respectively to obtain the mapped images and from which the identity mapping loss L Idt is calculated.

[0077] In the cycle consistency loss, the low-quality image x is input into the high-quality generator G L→H for enhancement, or the high-quality image y is input into the low-quality generator G H→L for restoration. If y is input into G L→H , we do not want the generated image G L→H (y) to change a lot, but want G L→H (y) to retain the features of y as much as possible, and learn the unit mapping of y, because y is a high-quality image. Similarly, inputting x into G H→L also does the same, we want G H→L (x) to be as similar to x as possible. This constraint can improve the ability of the GAN network to preserve image features such as color, which is called identity mapping loss L Idt , defined as:

[0078] L Idt =E x [||G H→L (x)-x||1]+E y [||GL→H (y) - y || 1] (6)

[0079] where x, y represent the original low-quality image and the original high-quality image, respectively; G L→H denotes the generator for generating high-quality images, G H→L denotes the generator for generating low-quality images.

[0080] Step 4, input the original high-quality image y and the enhanced image to the discriminator D H , and D H discriminates their authenticity, input the original low-quality image x and the generated low-quality image to the discriminator D L , and D L discriminates their authenticity, and calculate the adversarial loss L Adv from them;

[0081] L Adv = L GAN (G L→H , D H , x, y) + L GAN (G H→L , D L , x, y) (7)

[0082] L GAN (G L→H , D H , x, y) = E x [1 - (D H (G L→H (x))) 2 ] + E y [(D H (y) - 1) 2 ] (8)

[0083] L GAN (G H→L , D L , x, y) = E y [1 - (D L (G H→L (y))) 2 ] + E x [(D L (x) - 1) 2 ] (9)

[0084] where x, y represent the original low-quality image and the original high-quality image, respectively; G L→H denotes the generator for generating high-quality images, G H→L denotes the generator for generating low-quality images; D HD represents a discriminator for discriminating whether an input image is a high-quality image L D represents a discriminator for discriminating whether an input image is a low-quality image.

[0085] Step 5, based on the Adam algorithm and the principle of error back propagation, the gradient is obtained by optimizing the overall loss function, first fix the generator, update the parameters of the discriminator, then fix the discriminator, update the parameters of the generator, repeat the above process until the upper limit of the training round is reached:

[0086]

[0087] Wherein, λ1, λ2, α, β, γ and η respectively represent the weights corresponding to the cycle consistency loss L Cyc , identity mapping loss L Idt , illumination uniformity loss L IU , texture fidelity loss L TF , perceptual loss L PL and image smoothing loss L IS .

[0088] The present application carries out image enhancement experiment based on CORN-2 corneal confocal data set, introduces several SOTA methods with excellent performance in the field for comparison, including two traditional methods of CLAHE and DCP, four deep learning methods of NST, MSG-Net, EnlightenGAN and StillGAN, and compares and evaluates the enhancement effect of the method of the present application on low-quality medical images from the aspects of qualitative analysis and quantitative analysis.

[0089] The qualitative analysis of the present application mainly adopts the way of naked eye observation, and requires the observer to evaluate the medical image quality from the aspects of overall structure, illumination distribution and local texture details.

[0090] (1) Illumination distribution comparison

[0091] Firstly, the difference in illumination distribution between the original low-quality image and the image enhanced by applying each advanced method is compared. Figure 10As shown, the first and third rows are the original low-quality image and the enhanced images from various methods, while the second and fourth rows are magnified images of the area circled in red. The original image exhibits highly uneven illumination distribution. Applying CLAHE, DCP, NST, and MSG-Net to enhance the image provides limited improvement in illumination quality; the uneven illumination even degrades the image's texture. While EnlightenGAN preserves structural information relatively well, its ability to balance illumination uniformity only applies locally, becoming more pronounced the more uneven the illumination of the original image. The method presented in this invention achieves the best visual effect among all methods, with its enhanced image's illumination distribution being most suitable for visual observation. Compared to StillGAN, it demonstrates superior ability to preserve foreground texture details.

[0092] (2) Texture detail comparison

[0093] Secondly, the differences in texture detail between the original low-quality image and the image enhanced by various advanced methods are compared. For example... Figure 11 As shown, the first and third rows are the original low-quality image and the enhanced images from various methods, while the second and fourth rows are magnified images of the area circled in red. The texture details of the original image are rather blurry, and the texture structures of the enhanced images from CLAHE, DCP, NST, and MSG-Net exhibit significant distortion. Compared to EnlightenGAN and StillGAN, the method of this invention produces clearer foreground texture details and achieves better enhancement results.

[0094] (3) Noise artifact comparison

[0095] Finally, the differences in noise artifacts between the original low-quality image and the image enhanced by various state-of-the-art methods are compared. Figure 12 As shown, the areas marked by red boxes in the first and third rows of images exhibit uneven intensity, corneal scarring, or imaging spots. During enhancement, these areas should not be mistaken for nerve fiber structures, as this would hinder the enhancement of surrounding real nerve fibers and interfere with clinical diagnosis. It can be seen that all image enhancement methods other than the method of this invention fail to distinguish this noise, instead identifying it as nerve fibers and enhancing it accordingly. The method of this invention, however, can identify the difference between these two structures, avoiding noise enhancement and instead learning the information of real nerve fibers surrounding the noise. Furthermore, it removes the boundary artifacts present in StillGAN, as indicated by the red dashed box, further improving image quality.

[0096] Quantitative analysis adopts four commonly used no-reference quality evaluation indexes, image entropy, average gradient (AvG), Brisque, NIQE, and PIQE, to evaluate the quality of medical images.

[0097] Table 1

[0098]

[0099] Table 1 lists the entropy, average gradient AvG, Brisque, NIQE, and PIQE of the original low-quality image and the images enhanced by each advanced method. Overall, the quality of the image enhanced by the method of the present application can be considered the highest, as its AvG, Brisque, and PIQE indexes are the best, and the image entropy index is the second best. Especially PIQE, which is significantly reduced compared to the original image and other methods, means that the reconstruction performance of the method of the present application is higher, and the texture structure of the enhanced image is clearer.

[0100] Although these no-reference image quality evaluation indexes cannot fully represent the naked eye observation, they prove to some extent the advantages of the method of the present application in enhancing image quality, so further comparison of these enhancement methods based on downstream segmentation tasks is needed.

[0101] Enhancing low-quality images is often for subsequent image analysis, classification, and segmentation, and the better the enhancement effect, the more conducive to accurately completing these tasks; if the enhancement effect is poor and the image quality is still poor, it may lead to deviation or even error in diagnostic analysis. In order to confirm the influence of image enhancement on downstream analysis tasks, each enhanced image is input into the Cs-Net pre-trained image segmentation network to segment the corneal nerve fibers, and the segmentation results of the enhanced image and the original low-quality image are compared.

[0102] The segmentation results are shown in Figure 13 Overall, the segmentation effect of the enhanced image of basically all methods is better than that of the original image, which shows that various methods have certain enhancement effect. Among them, the method of the present application segments a more complete nerve fiber structure, and the continuity between some texture details is stronger, which is closest to the real label. It can be seen that the method of the present application can significantly improve the quality of the image, and the enhanced image can achieve good results in subsequent downstream segmentation tasks.

[0103] The quantitative analysis adopts six common indexes of AUC, pixel-by-pixel accuracy ACC (Accuracy), sensitivity SEN (Sensitivity), G-mean, Kappa coefficient and Dice similarity coefficient to evaluate the segmentation effect, and these evaluation indexes are positive indexes, that is, the greater the value is, the better the segmentation effect is.

[0104] Table 2 lists the AUC, ACC, SEN, G-mean, Kappa and Dice indexes of the original low-quality image and the images enhanced by each advanced method and input into the Cs-Net for segmentation. Overall, compared with the segmentation result of the original image, the images enhanced by the six methods have certain improvement, especially EnlightenGAN, StillGAN and the method of the application. Among the three methods with relatively good performance, the method of the application achieves the best result, and the five indexes except ACC are the highest. Among them, the AUC, SEN, G-mean, Kappa and Dice of the method of the application are improved by 3.1%, 7.1%, 5.3%, 3.1% and 3.3% respectively compared with the second-best StillGAN. The accuracy of nerve fiber structure segmentation is the basis for observing and diagnosing nervous system diseases, and the segmentation result reflects the effectiveness of enhancing image quality from the side.

[0105] Table 2

[0106]

[0107] The medical image enhancement method based on the multi-scale attention generative adversarial network provided by the above-mentioned embodiments of the application is trained in the confrontation of two generators and two discriminators, attention gates are added when the encoding layer is connected to filter invalid information, highlight important image regions and avoid redundant bottom feature accumulation; the decoding layer is expanded by designing the pyramid feature FPN to fuse features at different levels and the end output, so that the multi-scale, context global information is better utilized; finally, the convolution attention module is used to perform attention allocation on the fused features in the channel and spatial dimensions, and more attention is paid to key information during imaging. The method adds the uniform illumination term, the texture fidelity term, the perception loss term and the image smoothness, and strengthens the constraints on the generated image quality from multiple angles such as illumination distribution, texture details, deep semantic features and smoothness to improve the enhancement effect. The application trains the network by using the Adam algorithm, and the experimental results verify the feasibility and effectiveness of the application.

[0108] Although the embodiments of the application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the application, and the scope of the application is defined by the appended claims and their equivalents.

Claims

1. A medical image enhancement method based on a multi-scale attention generative adversarial network, characterized in that, Comprise the following steps: S1, construct generator G L→H and discriminator G L→H , for the enhanced image branch; construct the same generator G H→L and discriminator G H→L , for the restored image branch; S2, input the original low-quality image x into the generator G L→H to obtain an enhanced image again input into the generator G H→L to obtain a restored low-quality image input the original high-quality image y into the generator G H→L to obtain a low-quality image again input into the generator G L→H to obtain a restored high-quality image cycle-consistency loss L Cyc , illumination uniformity loss L IU , texture fidelity loss L TF , image smoothness loss L IS , and perceptual loss L PL ; S2.1, input the original low-quality image x and the original high-quality image y into the generator G H→L and G L→H to obtain a mapped image and from which the identity mapping loss L Idt is calculated; S2.2, Combine the original high-quality image y with the enhanced image Input to discriminator D H In the middle, by D H To determine their authenticity, compare the original low-quality image x with the generated low-quality image x. Input to discriminator D L In the middle, by D L Determine their authenticity and calculate the adversarial loss L from them. Adv ; S3, based on Adam algorithm and error back propagation principle, by optimizing the overall loss function to obtain gradient, first fixed generator, update the parameters of the discriminator; Then fixed discriminator, update the parameters of the generator; Repeat the above process until the upper limit of the training round.

2. The medical image enhancement method based on the multi-scale attention generative adversarial network according to claim 1, characterized in that, In the step S1, the generator takes U-Net based on convolution residual module as the main body, adds attention gate when jumping connection in the encoding layer to filter invalid information, highlights important image area, and avoids redundant bottom feature accumulation; Design pyramid feature FPN to expand the decoding layer, fuse features of different levels and end output, and better utilize multi-scale and context global information; Finally, the convolution attention module is used to allocate attention to the fused features in the channel and spatial dimensions, and more attention can be paid to the key information during imaging. 3.The medical image enhancement method based on multi-scale attention generative adversarial network according to claim 1, characterized in that, In the step S2, the discriminator is a multi-scale neural network composed of multiple convolution modules; In the step S2, a cycle-consistent loss L is calculated Cyc , an illumination uniformity loss L IU , a texture fidelity loss L TF , an image smoothness loss L IS and a perceptual loss L PL , Comprise: L Cyc = E x [‖G H→L (G L→H (x))-x‖1]+E y [‖G L→H (G H→L (y))-y‖1] (1) The (1) formula, wherein x, y respectively represent the original low-quality image and the original high-quality image; G L→H G represents a generator for generating a high-quality image H→L G represents a generator for generating a low-quality image (2) wherein, E global [·]、 respectively represent the global average illumination of the input image and the local region average illumination of p x p size; represent a bicubic interpolation up-sampling operation to eliminate the scale difference; (3) where, K denotes the number of local regions divided in each image; y i , x i denote the i-th local region of the high-quality image y and the low-quality image x, respectively, G L→H (x) i , G H→L (y) i denote the i-th local region of the two generated images, respectively; and denote the covariance matrix between the i-th region of the original image and the i-th region of the corresponding generated image, are the corresponding standard deviation matrixes of y i , G H→L (y) i , x i , G L→H (x) i , respectively; c is a small constant used to avoid numerical instability. (4) wherein, represents the feature map extracted by the jth layer of convolution after the ith layer of pooling of the pre-trained network, and H and W are the size of the image described by the intensity function. (5) In the formula, wherein, f(x,y) is the intensity function of the given image, H and W are the size of the image described by the intensity function.

4. The medical image enhancement method based on the multi-scale attention generative adversarial network according to claim 1, characterized in that, In the step S2.2, the adversarial loss L is calculated Adv : L Adv = L GAN (G L→H , D H , x, y) + L GAN (G H→L , D L , x, y) (7) L GAN (G L→H ,D H ,x,y)=E x [1-(D H (G L→H (x))) 2 ]+E y [(D H (y)-1) 2 ] (8) L GAN (G H→L ,D L ,x,y)=E y [1-(D L (G H→L (y))) 2 ]+E x [(D L (x)-1) 2 ] (9). 5.The medical image enhancement method based on multi-scale attention generative adversarial network according to claim 1, characterized in that, In the step S3, based on Adam algorithm and error back propagation principle, by optimizing the overall loss function to obtain gradient, first fixed generator, update the parameters of the discriminator; Then fixed discriminator, update the parameters of the generator; Repeat the above process until the upper limit of the training round. In the formulas (10) and (11), λ1, λ2, α, β, γ and η respectively represent weights corresponding to cycle-consistent loss L Cyc , identity mapping loss L Idt , illumination uniformity loss L IU , texture fidelity loss L TF , perceptual loss L PL and image smoothing loss L IS .

Images