Improved cycle gan based ultrasound image speckle noise reduction method

Abstract

An improved ultrasound image speckle noise reduction method based on cycle generative adversarial network, obtains GAD noise reduction image of medical ultrasound image data set; the GAD noise reduction image is scored using the proposed ENR noise reduction image quality evaluation index, and the image used as the training set and the test set is screened out;Introduce the cycle consistency structure, build the cycle generative adversarial network model CycleGAN for noise reduction, and use two sets of generators and discriminators for bidirectional training. Training is carried out using the training set, and the appropriate CycleGAN model is determined according to the checkpoint and the noise reduction result;In the online stage, the test set image is denoised by the trained CycleGAN model, and the ENR is used to evaluate the image noise reduction quality. The present application can obtain excellent noise reduction effect for medical ultrasound images contaminated by speckle noise.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of image processing, and particularly relates to an improved medical ultrasonic image speckle noise denoising method based on a cycle generative adversarial network (CycleGAN). BACKGROUND

[0002] Compared with other clinical imaging methods, an ultrasonic image is susceptible to clutter interference, so that noise mainly in the form of speckle noise appears in the image, which affects the resolution of the obtained image and the judgment of a lesion. An anisotropic diffusion based on Gabor transform (GAD) improves an edge detector of anisotropic diffusion by using Gabor transform, so as to improve the distinguishability of a tissue edge and a noise region, thereby guiding a diffusion process and improving denoising performance. However, due to the limitation of the number of cycles, the denoising level of the image is unstable, and the denoising effect of part of the images is good, but the fine details and edges of part of the images disappear, which leads to the degradation of denoising performance. In recent years, a cycle generative adversarial network is widely applied to the style transfer of natural images and medical images. Compared with other traditional methods and simple neural networks, the design of the bidirectional mapping structure makes the conversion performance more powerful. The application relates to an improved ultrasonic image speckle noise denoising method based on a cycle generative adversarial network, and high-quality images after GAD denoising are used as a noise-free data set, so as to improve the denoising performance of CycleGAN. SUMMARY

[0003] In order to solve the defects that a traditional image denoising method cannot effectively reduce speckle noise and a traditional CycleGAN cannot denoise ultrasonic images due to the inability to collect noise-free ultrasonic images, the application provides an improved ultrasonic image speckle noise denoising method based on a cycle generative adversarial network. The medical ultrasonic image polluted by speckle noise also has excellent denoising effect.

[0004] The application is implemented by the following technical scheme:

[0005] The application relates to an improved ultrasonic image speckle noise denoising method based on a cycle generative adversarial network, and the method comprises the following steps:

[0006] Step 1) obtaining a medical ultrasonic image data set and processing the ultrasonic image;

[0007] Step 2) obtaining a GAD denoising image;

[0008] Step 3) scoring the GAD denoising image by using an ENR denoising image quality evaluation index, and screening out images used as a training set and a test set;

[0009] Step 4) Make a data set and pre-process the data set, the purpose is to divide the training set and the test set, the training set is used for training the network model, and the test set is used for testing the performance of the model;

[0010] Step 5) Build a network model: introduce a loop consistency structure, build a cycle generative adversarial network model CycleGAN for noise reduction, and use two sets of generators and discriminators for bidirectional training.

[0011] The bidirectional training refers to: for the denoising task, in the forward mapping process from the X domain, i.e. the noisy image domain to the Y domain, i.e. the noise-free image domain, the generator G and the discriminator are used to map the noisy image in the X domain to the noise-free image in the Y domain; in the reverse mapping process from the Y domain, i.e. the noise-free image domain to the X domain, i.e. the noisy image domain, the generator F and the discriminator are used to map the noise-free image in the Y domain to the noisy image in the X domain, and training is carried out in both mapping directions to obtain the bidirectional mapping corresponding to the noisy image in the X domain and the noise-free image in the Y domain.

[0012] Step 6) Training using the training set, and determining the appropriate CycleGAN model according to the checkpoint and the denoising result;

[0013] Step 7) Using the trained CycleGAN model, denoising the test set image, and using ENR to evaluate the image denoising quality.

[0014] Technical effects

[0015] The present application aims at the problem that the noise-free image cannot be obtained from the ultrasound image, uses the GAD algorithm, uses the image with good denoising effect after processing as the "noise-free data set" required by the network, learns the advantages of the GAD algorithm, and improves the data set quality. Compared with the traditional method and the simple neural network, the cycle generative network uses the double-direction general mapping to convert and denoise between the noise data domain with speckle noise and the noise-free data domain. At the same time, CycleGAN is mostly applied to the conversion task of ideal target image, and the present application proposes to apply it to the denoising task of medical ultrasound image without natural noise-free image.

[0016] Compared with the prior art, the present application can improve the denoising effect from the root of the data set, highlight the contrast between the lesion area and the background area, and retain the fine details of the lesion edge. At the same time, CycleGAN is mostly applied to the conversion task of ideal target image, and the present application proposes to apply it to the denoising task of medical ultrasound image without natural noise-free image, so that its application is more widely. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The flowchart of the present application;

[0018] Figure 2 This is a schematic diagram of the CycleGAN network model;

[0019] Figure 3 A schematic diagram of the CycleGAN generator and discriminator architecture;

[0020] Figure 4 The image shown is a rendering of an example. Detailed Implementation

[0021] like Figure 1 As shown in the figure, this embodiment relates to a method for reducing speckle noise in ultrasound images based on CycleGAN, which includes the following steps:

[0022] Step 1: Obtain a medical ultrasound image dataset and process the ultrasound images. Specifically, obtain ultrasound images of breast tumors and crop and scale the noisy ultrasound images to 128*128 pixels.

[0023] Step 2: Perform cyclic filtering using the GAD algorithm to obtain the GAD-denoised image;

[0024] The specific operation of the GAD algorithm includes: inputting the ultrasound image into the GAD algorithm; GAD uses Gabor transform to capture edge directionality, thereby deriving a Gabor-based robust edge detector; and then embedding the Gabor-based edge detector into AD to guide the diffusion process. During cyclic denoising in GAD, the absolute error between the denoised image and the previous image is calculated in each round. The loop ends when the number of rounds is greater than or equal to a set number of rounds and the absolute error is less than or equal to a set threshold.

[0025] Step 3: Based on the GAD denoised images obtained in Step 2, score the GAD denoised images using the proposed ENR denoised image quality evaluation index, and select images to be used as training and test sets.

[0026] The aforementioned ENR noise reduction image quality evaluation index Where: σ e σ represents the standard deviation of the edge regions in the image. i σ represents the standard deviation of the target region in the image. b The standard deviation of the background region in the image is used to automatically calculate the ENR score by sequentially selecting the target region, background region, and edge region.

[0027] Step 4: Create a dataset and preprocess it. The purpose of this is to divide the dataset into a training set and a test set. The training set is used to train the network model, while the test set is used to test the performance of the model.

[0028] The making data set includes selecting 60 images with higher scores, rotating the noisy ultrasound images and GAD denoising images by 90°, 180° and 270° respectively, and mirror flipping, and the processed noisy ultrasound images and GAD denoising images are used as training set X and training set Y respectively. 10 noisy ultrasound images with lower scores are selected as the test set.

[0029] The data set image uses the breast tumor ultrasound imaging public data set provided by Baheya Women's Cancer Early Detection and Treatment Hospital in Cairo, Egypt, and processes all images to 128*128 size. The GAD is used for denoising the ultrasound image, and the processing is ended when the denoising round is greater than or equal to 100 rounds and the absolute value error between the processed image and the previous image is less than or equal to 0.005. The ENR is used to score the denoised image, and the top 60 images with higher scores are selected as the training set, and the top 10 images with lower scores are selected as the test set. The data of the training set is processed by image including random rotation 90°, random rotation 180°, random rotation 270° and mirror flipping data enhancement method to expand the data amount and improve the generalization of training.

[0030] Step 5, construct a cycle generative adversarial network model for denoising as shown in Figure 2 The cycle generative adversarial network model for denoising includes a generator and a discriminator, wherein: the generator reflects and fills the input image, then performs three convolution layers, an activation function layer, and an instance normalization layer to down-sample and extract features, and then enters a residual neural network module; and the discriminator compresses the input image into a 30*30*1 size matrix, and then evaluates the effect of the generated image by calculating a loss function.

[0031] As shown in Figure 3 The generator uses a residual neural network, specifically including: 9 cascaded same residual neural network modules, after the feature map passes through the last residual neural network module, it passes through two layers of fractional step convolution layers, an instance normalization layer, and an activation function layer for upsampling, and finally passes through a reflection padding layer, a convolution layer and a Tanh activation function to output the generated image generated by the generator.

[0032] The discriminator uses a PantchGAN discriminator, specifically including: five convolutional neural network layers, an instance normalization layer and an activation function, wherein: the five convolutional neural network layers gradually compress the input image into a 30*30 feature matrix, and finally make each point in the feature matrix represent a 70*70 size receptive field. The input image size is 256*256*3, after passing through five convolutional layers, in order to facilitate the discriminator to evaluate the image quality, the output channel number of the last convolutional neural network layer is set to 1, and the output result is true or false.

[0033] The loss function comprises: a cycle consistency loss function, a cycle consistency loss function and an identity loss function corresponding to two different groups of generators and discriminators, so that the image generated by the generator is as similar as possible to the image of the target domain, so that the discriminant cannot distinguish between the fake image generated by the generator and the real image of the target domain; ensure the consistency of the generated image, that is, the generated image should have a correlation with the input image, and the generator cannot just generate an image that can deceive the discriminator; and fit the generated image to the probability distribution of the image in the target domain, when the input image already conforms to the probability distribution of the image in the target domain, the generator should not change any features of the image, so as to prevent the deviation of color and pixel position of the generated result.

[0034] The adversarial loss function The adversarial loss is obtained, wherein: L GAN (G,D Y ,X,Y) represents the adversarial loss function, E[·] represents the expected value, D Y represents the discriminator network, G represents the generator network, X represents the data set X (original ultrasound image), and Y represents the data set Y (high-quality GAD denoising image). The corresponding adversarial loss expression of the reverse mapping is the same as that of the forward mapping, only the corresponding generator and discriminator are changed, and the total adversarial loss is obtained by adding the adversarial losses of the bidirectional mapping.

[0035] The cycle consistency loss function Wherein: L CYCLE (G,F) represents the cycle consistency loss function, E[·] represents the expected value, G represents the generator network, F represents the generator network, X represents the data set X (original ultrasound image), and Y represents the data set Y (high-quality GAD denoising image). The result generated by the generator G is taken as a new input through the generator F, and the absolute value of the difference between the output image of the generator F and the original input is calculated. The reverse mapping is the same, and the result generated by the generator F is taken as a new input through the generator G, and the loss is calculated. The total cycle consistency loss is obtained by adding the cycle consistency losses of the bidirectional mapping.

[0036] The identity loss function Wherein: L IDENTITY(G, F) represents the identity loss function, E[·] represents the expected value, G represents the generator network, F represents the generator network, X represents the data set X (ultrasound image), Y represents the data set Y (high-quality GAD denoising image). The difference value is calculated after the image in the X domain is processed by the generator F, and then the difference value is calculated after the image in the Y domain is processed by the generator G. For an ideal generator deep network, the image in the X domain should not change after being processed by the generator F, and the image in the Y domain should not change after being processed by the generator G. The total identity loss is obtained by adding the two-way identity loss.

[0037] Step 6, training using the training set, determining the appropriate CycleGAN model according to the checkpoint and the denoising result, specifically: using the Adam optimizer with linearly decaying learning rate, the initial learning rate is set to 0.0002. And set the checkpoint, save the model every certain number of epochs during training, to find the best number of epochs for different data sets. The MSELoss function is used to calculate the adversarial loss of the generator and the discriminator, and the L1 Loss function is used to calculate the cycle consistency loss and the identity loss. After calculating the loss function of each part, add them together as the total loss function.

[0038] Step 7, using the trained CycleGAN model, denoising the test set image, and using ENR to evaluate the image denoising quality, as shown in Figure 4 The present application uses ENR to evaluate the denoising result. Compared with the prior art, the CycleGAN enhances the tumor boundary and better preserves the details without over-smoothing, and achieves excellent results with an average ENR index of 28.47±6.18 on 10 test data.

[0039] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application. The protection scope of the present application is subject to the claims and is not limited by the above specific embodiments. Each implementation within the scope is subject to the constraints of the present application.

Claims

1. A CycleGAN-based speckle noise reduction method for ultrasound images, characterized in that, In the offline stage, the GAD algorithm is used to denoise the collected ultrasound images, and the ENR is used to score and screen the denoised images, and the preprocessed training data set is generated to train the CycleGAN network; in the online stage, the trained CycleGAN network is used to output the denoised result image according to the input noisy original image; The GAD algorithm specifically includes: inputting the ultrasound image into the GAD algorithm, using Gabor transformation to capture edge directionality, thereby deriving a Gabor-based robust edge detector, then embedding the Gabor-based edge detector into the AD to guide the diffusion process, and when the cycle denoising is performed by the GAD, the absolute value error of the denoised image and the image of the last round is calculated every round, and when the cycle round is greater than or equal to the set round and the absolute value error is less than or equal to the set threshold, the cycle is ended; The cycle generative adversarial network model includes a generator and a discriminator, wherein: the generator performs reflective padding on the input image, then performs three convolution layers, an activation function layer, and an instance normalization layer to down-sample and extract features, and then enters a residual neural network module, and the discriminator compresses the input image into a 30*30*1 size matrix, and then evaluates the effect of the generated image by calculating a loss function; The generator uses a residual neural network, specifically including: 9 cascaded same residual neural network modules, after the feature map passes through the last residual neural network module, it passes through two layers of fractional step convolution layers, an instance normalization layer, and an activation function layer for upsampling, and finally passes through a reflective padding layer, a convolution layer, and a Tanh activation function to output the generated image generated by the generator; The discriminator uses a PantchGAN discriminator, specifically including: five convolutional neural network layers, an instance normalization layer, and an activation function, wherein: the five convolutional neural network layers gradually compress the input image into a 30*30 feature matrix, and finally make each point in the feature matrix represent a 70*70 size receptive field, the input image size is 256*256*3, after passing through the five convolutional layers, in order to facilitate the discriminator to evaluate the image quality, the output channel number of the last convolutional neural network layer is set to 1, and the output result is true or false.

2. The CycleGAN-based ultrasound image speckle noise reduction method of claim 1, wherein, It includes: Step 1) Obtain a medical ultrasound image data set, and process the ultrasound image; Step 2) Obtain a GAD denoised image; Step 3) Score the GAD denoised image using the proposed ENR denoised image quality evaluation index, and select the images used as the training set and the test set; Step 4) Make a data set and pre-process the data set, the purpose is to divide the training set and the test set, the training set is used to train the network model, and the test set is used to test the performance of the model; Step 5) Build a network model: introduce a cycle consistency structure, build a cycle generative adversarial network model CycleGAN for denoising, and use two sets of generators and discriminators for bidirectional training; The bidirectional training refers to: for the denoising task, in the forward mapping process from the X domain, i.e. the noisy image domain to the Y domain, i.e. the noise-free image domain, the generator G and the discriminator are used to map the noisy image in the X domain to the noise-free image in the Y domain; in the reverse mapping process from the Y domain, i.e. the noise-free image domain to the X domain, i.e. the noisy image domain, the generator F and the discriminator are used to map the noise-free image in the Y domain to the noisy image in the X domain, and training is performed in both mapping directions to obtain bidirectional mapping corresponding to the noisy image in the X domain and the noise-free image in the Y domain; Step 6) training using the training set, determining the appropriate CycleGAN model according to the checkpoint and the denoising result; Step 7) using the trained CycleGAN model to denoise the test set images, and using ENR to evaluate the image denoising quality.

3. The CycleGAN-based ultrasound image speckle noise reduction method of claim 1 or 2, characterized in that, The ENR noise reduction image quality evaluation index Wherein: sigma e The standard deviation of the edge region in the image, sigma i The standard deviation of the target region in the image, sigma b The standard deviation of the background region in the image, sequentially selecting the target region, the background region and the edge region, and automatically calculating the ENR score.

4. The CycleGAN-based ultrasound image speckle noise reduction method of claim 1, wherein, The data set is made by selecting 60 images with higher scores, rotating the noisy ultrasound images and GAD denoised images by 90°, 180° and 270° respectively, and mirror flipping, and the processed noisy ultrasound images and GAD denoised images are used as training set X and training set Y respectively; 10 noisy ultrasound images with lower scores are selected as the test set.

5. The CycleGAN-based ultrasound image speckle noise reduction method of claim 2, wherein, The loss function includes: through the cycle consistency loss function of the corresponding two different generators and discriminators, the cycle consistency loss function and the identity loss function, the generated image is as similar as possible to the image in the target domain, so that the discriminator cannot distinguish between the fake image generated by the generator and the real image in the target domain; ensure the consistency of the generated image, that is, the generated image should have a correlation with the input image, and the generator cannot just generate an image that can deceive the discriminator; and the generated image is fitted to the probability distribution of the image in the target domain, when the input image already conforms to the probability distribution of the image in the target domain, the generator should not change any features of the image, so as to prevent the deviation of color and pixel position of the generated result.

6. The CycleGAN-based ultrasound image speckle noise reduction method of claim 5, wherein, The adversarial loss function The adversarial loss is obtained, wherein: L GAN (G, D Y , X, Y) represents the adversarial loss function, E[·] represents the expected value, D Y represents the discriminator network, G represents the generator network, X represents the data set X (original ultrasound image), Y represents the data set Y (high-quality GAD denoising image); the corresponding adversarial loss expression of the reverse mapping is the same, only the corresponding generator and discriminator are changed, and the total adversarial loss is obtained by adding the adversarial losses of the two-way mapping. The cycle consistency loss function Wherein: L CYCLE (G, F) represents a cycle consistency loss function, E[·] represents an expected value, G represents a generator network, F represents a generator network, X represents a data set X (original ultrasound image), Y represents a data set Y (high-quality GAD denoising image); the result generated by the generator G is taken as a new input through the generator F, and the absolute value of the difference between the output image of the generator F and the original input is calculated; the reverse mapping is the same, the result generated by the generator F is taken as a new input through the generator G, and the loss is calculated; the cycle consistency loss of the bidirectional mapping is added to obtain the total cycle consistency loss. The identity loss function Wherein: L IDENTITY (G, F) represents the identity loss function, E[·] represents the expected value, G represents the generator network, F represents the generator network, X represents the data set X (ultrasound image), Y represents the data set Y (high-quality GAD denoising image); the difference value is calculated after the image in the X domain is subjected to the generator F, then the difference value is calculated after the image in the Y domain is subjected to the generator G, for an ideal generator deep network, the image in the X domain should not change after being subjected to the generator F, and the image in the Y domain should not change after being subjected to the generator G, and the total identity loss is obtained after adding the two-way identity losses.

7. The CycleGAN-based ultrasound image speckle noise reduction method of claim 2, wherein, The training refers to: using the Adam optimizer with linearly decaying learning rate, the initial learning rate is set to 0.0002, and a checkpoint is set, the model is saved every certain number of epochs during the training process, to find the best number of epochs for different data sets, the MSELoss function is used to calculate the adversarial loss of the generator and the discriminator, the L1 Loss function is used to calculate the cycle consistency loss and the identity loss, and after calculating the loss function of each part, the total loss function is added.

Images