General method and device for image quality enhancement based on cycle consistency loss

Abstract

A general method and device for image quality enhancement based on cycle consistency loss, which does not require identification of task categories, uses a model to universally solve three endoscopic image quality enhancement problems: smoke removal, illumination enhancement, and color correction. The steps are simple and the space occupied by the model is greatly reduced. While having the universality of not needing to distinguish task categories, it can still achieve the image enhancement effect of training for a single task. The method includes: (1) introducing cycle consistency loss through unsupervised adversarial training of two processes, forward loop and backward loop, and realizing network training using three unpaired real endoscopic images; (2) using a general model training strategy, using three types of image quality defect data to update network parameters three times in one iteration; (3) using the forward generator network obtained during the training process to universally handle smoke removal, illumination enhancement, and color correction tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a general method for enhancing image quality based on cycle consistency loss, and a general device for enhancing image quality based on cycle consistency loss. Background Art

[0002] Endoscopic imaging has many defects, and most existing methods can only solve a specific endoscopic image quality enhancement problem. Usually, three separately trained models are used to solve the three endoscopic image quality enhancement problems of smoke removal, illumination enhancement, and color correction. This is cumbersome and the model takes up space. Summary of the invention

[0003] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide a general method for image quality enhancement based on cycle consistency loss, which does not require the identification of task categories and uses one model to universally solve the three endoscopic image quality enhancement problems of smoke removal, illumination enhancement, and color correction. The steps are simple and the space occupied by the model is greatly reduced. While it has the versatility of not needing to distinguish task categories, it can still achieve the image enhancement effect of training for a single task.

[0004] The technical solution of the present invention is: this general method for image quality enhancement based on cycle consistency loss comprises the following steps:

[0005] (1) Through unsupervised adversarial training of the forward and backward cycles, the cycle consistency loss is introduced to achieve network training using three unpaired real endoscopic images;

[0006] (2) Using a general model training strategy, the network parameters are updated three times using three types of image quality defect data in one iteration;

[0007] (3) The forward generator network obtained during the training process is used to universally handle smoke removal, lighting enhancement, and color correction tasks.

[0008] The present invention introduces cycle consistency loss through unsupervised adversarial training of two processes, forward loop and backward loop, and realizes network training using three unpaired real endoscopic images. A universal model training strategy is used to update network parameters three times using three types of image quality defect data in one iteration. Finally, the forward generator network obtained in the training process can be used to universally handle smoke removal, illumination enhancement, and color correction tasks. Not only does it eliminate the step of low-quality image classification, but the model occupies only one-third of the space of a single training model. It has the universality of not needing to distinguish task categories, and can still achieve the image enhancement effect of single training for one task.

[0009] A general apparatus for image quality enhancement based on cycle consistency loss is also provided, which comprises:

[0010] The network training module is configured to conduct unsupervised adversarial training through two processes, forward loop and backward loop, and introduce cycle consistency loss to achieve network training using three unpaired real endoscopic images;

[0011] A network parameter update module, which is configured to use a general model training strategy to update the network parameters three times using three types of image quality defect data in one iteration;

[0012] The defect handling module is configured to utilize the forward generator network obtained during the training process to generally handle smoke removal, lighting enhancement, and color correction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flowchart of a general method for image quality enhancement based on cycle consistency loss according to the present invention.

[0014] Figure 2 is the generator network structure.

[0015] Figure 3 is the discriminator network structure.

[0016] Figure 4 The activation function is shown.

[0017] Figure 5 It is a general framework pseudo code for endoscopic image quality enhancement.

[0018] Figure 6 A quantitative comparison of the endoscopic image quality enhancement effect is shown. DETAILED DESCRIPTION

[0019] like Figure 1 As shown, this general method for image quality enhancement based on cycle consistency loss includes the following steps:

[0020] (1) Through unsupervised adversarial training of the forward and backward cycles, the cycle consistency loss is introduced to achieve network training using three unpaired real endoscopic images;

[0021] (2) Using a general model training strategy, the network parameters are updated three times using three types of image quality defect data in one iteration;

[0022] (3) The forward generator network obtained during the training process is used to universally handle smoke removal, lighting enhancement, and color correction tasks.

[0023] The present invention introduces cycle consistency loss through unsupervised adversarial training of two processes, forward loop and backward loop, and realizes network training using three unpaired real endoscopic images. A universal model training strategy is used to update network parameters three times using three types of image quality defect data in one iteration. Finally, the forward generator network obtained in the training process can be used to universally handle smoke removal, illumination enhancement, and color correction tasks. Not only does it eliminate the step of low-quality image classification, but the model occupies only one-third of the space of a single training model. It has the universality of not needing to distinguish task categories, and can still achieve the image enhancement effect of single training for one task.

[0024] Preferably, the network structure of the method includes a generator and a discriminator;

[0025] like Figure 2 As shown in the figure, the generator consists of an encoder, a converter and a decoder. The image is resized to 256×256 and then input into the encoder composed of three convolution modules; the convolution module includes a convolution layer, an instance normalization layer and a ReLU activation function layer; the convolution layer in the first convolution module uses a mirror edge padding of size 3, a step size of 1, and a convolution kernel size of 7×7×64; the convolution layer edge padding size used in the second convolution module is 1, the step size is 2, and the convolution kernel size is 3×3×128; the convolution layer edge padding size used in the third convolution module is 1, the step size is 2, and the convolution kernel size is 3×3×256; The encoder consists of two deconvolution modules and one convolution module; the first deconvolution module uses a deconvolution layer with an edge padding size of 1, a stride of 2, and a deconvolution kernel size of 3×3×128; the second deconvolution module uses a deconvolution layer with an edge padding size of 1, a stride of 2, and a deconvolution kernel size of 3×3×64; both deconvolution layers are followed by an instance normalization layer and a ReLU activation function layer; the convolution layer used in the last convolution module is the same as the first convolution module in the encoder, but with 64 input channels and 3 output channels; the output is obtained after instance normalization and Tanh activation function; the converter consists of 9 Resnet Each Resnet Block has the same network structure, and uses two convolutional modules combined with skip connections. The convolutional layer used in the first convolutional module uses a mirror edge padding of size 1, a step size of 1, and a convolution kernel size of 3×3×256, followed by an instance normalization layer and a ReLU activation function layer. The second convolutional module uses the same structure but does not include a ReLU activation function layer.

[0026] like Figure 3As shown in the figure, the discriminator adopts a 70×70 PatchGAN, which consists of five convolution modules. The convolution layer parameters used in the five convolution modules are the same, with a step size of 2, an edge padding size of 1, a convolution kernel size of 4×4, and the number of output channels of 64, 128, 256, 512 and 1 respectively. Instance normalization and Leaky ReLU function activation are performed, where the slope of Leaky ReLU is 0.2, and instance normalization is not used in the first convolution module.

[0027] The purpose of network training is to transform a class of images with image quality defects into Transformed into high visual quality image sets with clear fields of view like normal endoscopic images

[0028] Preferably, given an endoscopic image I with image quality defects defect , input to the generator G ideal Generate a high-quality image I ideal ′, discriminator D ideal By comparing with a high-quality endoscopic image ideal Compare to determine the generated high-quality image I ideal ′ is true or false; generator G defect The purpose is to generate an image with image quality defects I defect ″, discriminator G defect Used to judge I defect Is it a real image with endoscopic image quality defects? defect ; Then for high-quality endoscopic image I ideal Using the same method, first pass the generator G defect Generate an image with quality defects I defect ′, and then through the generator G ideal From I defect ′Restore high-quality endoscopic image I ideal ″, respectively using the discriminator D defect and D ideal To make a judgment; the above process is respectively a forward loop and a backward loop, which are expressed as follows:

[0029]

[0030]

[0031] Maintaining the balance between the generator and the discriminator by applying an adversarial loss function while enhancing images with image quality defects;

[0032] The least squares loss is used instead of the traditional negative log-likelihood loss function to enhance the stability of training and generate higher image quality results. The forward loop adversarial loss and the backward loop adversarial loss are expressed as follows:

[0033]

[0034]

[0035] Preferably, a cycle consistency loss is introduced so that the content of the output high-quality endoscopic image is consistent with the input image. The cycle consistency loss is expressed as follows:

[0036]

[0037] The complete objective loss function is a combination of adversarial loss and cycle consistency loss, expressed as follows:

[0038]

[0039] The optimization goal is:

[0040]

[0041] The activation function is an important component of deep neural networks. Its main function is to complete the nonlinear transformation from the output signal of the previous node to the input signal of the next node. On the one hand, the ability of neural networks that only use linear transformation to learn complex function mapping is limited. Introducing activation functions to add nonlinear transformations to the network can effectively improve the ability of neural network models to solve complex problems. On the other hand, activation functions can effectively perform data normalization, limit data expansion and prevent overflow.

[0042] Sigmoid is the most basic activation function, which can map the input value to the output in the range of (0,1). Figure 4 (a) shows the function graph, and the expression is as follows:

[0043]

[0044] The non-zero output center of the Sigmoid function can easily cause the parameters to be updated in only one direction, and the exponential operation involved in the denominator will result in high computational costs. Figure 4 As shown in (b), since most of the values ​​of the derivatives of the sigmoid function are close to zero, the gradient vanishing problem is prone to occur during the back propagation process. The tanh function is similar to the sigmoid function, but the difference is that it maps the output to the range of (-1,1), solving the non-zero mean problem of the sigmoid function. Its image is shown in Figure 4 (c) is shown in the following expression:

[0045]

[0046] Combination Figure 4 (d) shows the function graph of its derivative. The tanh function still has the problem of exponential operation and gradient disappearance. The most widely used function is the ReLU function, such as Figure 4 (e) is shown in the following expression:

[0047] ReLU(x)=max(0,x) (2.9)

[0048] Because the derivative of the ReLU function in the non-negative interval is always a constant 1, the gradient vanishing problem is avoided. And its calculation is simple, making the convergence speed much faster than the sigmoid function and tanh function. However, the gradient of ReLU in the negative interval is zero, which may cause some neurons not to be activated by any data and the corresponding parameters cannot be updated. To address this problem, Maas et al. proposed the LeakyReLU function, such as Figure 4 (f), its expression is as follows:

[0049] LeakyReLU(x)=max(ax,x) (2.10)

[0050] It adds a small slope α in the negative interval, so that the gradient in the negative interval is no longer zero, avoiding the occurrence of inactivated neurons.

[0051] Combining the advantages and disadvantages of the above four activation functions, different activation functions are used in the generator and discriminator. The generator in the general framework for endoscopic image quality enhancement proposed in this chapter uses 9 ResNet Blocks, so the model calculation and convergence speed are required to be high. Preferably, the ReLU activation function is used in the generator, and the tanh function is used to map the result to the range of (-1,1); the LeakyReLU function with a slope of 0.2 is used in the discriminator.

[0052] The optimizer is mainly used to update and calculate the parameters in the network model to minimize (or maximize) the loss function. It is mainly divided into three categories: gradient descent, momentum optimization, and adaptive learning rate. Among them, gradient descent is the most classic optimization method, and there are three main types. Batch gradient descent (BGD) is the most primitive form of gradient descent. Since all samples need to be traversed in each iteration to update the parameters, the training process is very slow. Stochastic gradient descent (SGD) uses only one sample to update the parameters in each iteration, which greatly speeds up the training speed. However, since the loss function on a single sample data is only randomly optimized, it cannot fully represent the performance of the loss function on the complete data, so it may converge to the local optimum and deviate from the global optimal direction. Mini-batch gradient descent (MBGD) combines the advantages of batch gradient descent and stochastic gradient descent. It uses a part of the samples to update the parameters in each iteration, which can effectively reduce the variance when updating the parameters and obtain stable convergence results. Its introduction of matrix operations can speed up the training speed. Momentum optimization makes improvements on the basis of gradient descent, aiming to reduce the swing amplitude of the gradient direction and speed up the convergence of the model. The momentum algorithm accumulates the previous gradient descent direction and combines it with the gradient direction of the current point to obtain the final gradient update direction. The Nesterov momentum algorithm (NAG) can predict the gradient direction of the next step, and then use this predicted value to update the parameters of the current position based on the momentum method. The adaptive learning rate method optimizes the training speed of the model by updating the learning rate. The Adagrad algorithm can dynamically adjust the update amplitude of the learning rate for parameters of different frequencies. However, due to the introduction of the cumulative sum of the squared gradient value in the denominator, it is easy to cause a sharp drop in the gradient when the number of iterations increases. The RMSprop algorithm has made improvements to this defect, modifying the cumulative sum of the squared gradient value in the denominator to an exponentially weighted moving average. The Adadelta algorithm has also made the same improvement, but it is no longer necessary to set a global learning rate. The Adam algorithm uses the first-order moment estimation and second-order moment estimation of the gradient to dynamically adjust the learning rate of each parameter. After bias correction, the parameter update is relatively stable because the range of the learning rate can be determined at each iteration.

[0053] Preferably, the Adam optimizer is selected as the optimizer of the image quality enhancement framework.

[0054] The present invention uses a general unsupervised adversarial training model to simultaneously restore images with smoke, uneven lighting, and color distortion. The specific training strategy pseudo code is as follows: Figure 5As shown. The method is mainly divided into a training phase and a testing phase. In the training phase, instead of simply mixing three data with different endoscopic image quality defects into one dataset for training, they are still treated as three separate datasets. In each iteration, the generator and discriminator parameters are updated in turn using smoke images, uneven illumination images, and color distorted images, that is, the parameters need to be updated three times in one iteration. Such a training strategy is conducive to balancing the response of the general generative adversarial network to different endoscopic image quality enhancement tasks.

[0055] During the training process, the generator network and the discriminator network are continuously trained adversarially, and their parameters are updated alternately. The generator parameters are fixed for a period of time to optimize the discriminator parameters, and the generator parameters are modified for another period of time while the discriminator parameters are kept fixed. After repeated adversarial learning, the generator and the discriminator finally reach a dynamic balance, at which point the discriminator cannot distinguish between the real high-quality endoscopic images and the enhanced images generated by the generator. In the test phase, the forward generator network G obtained in the training phase is used to train the discriminator. ideal Directly restore the input endoscopic images with different defects.

[0056] In order to further illustrate the effect of this method on image quality enhancement, Figure 6 The quantitative comparison of the model and the separately trained models Zhu-S, Zhu-I, and Zhu-C in terms of endoscopic image quality enhancement is given in the paper, mainly listing the average values ​​of four evaluation indicators SSIM, PSNR, FSIMc, and QSSIM on 100 test images with gold standards. Figure 6 It can be seen that the average effect of our method CG-3 is comparable to that of Zhu et al.'s method when trained alone, and even has a slight advantage in the tasks of illumination enhancement and color correction.

[0057] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be: ROM / RAM, disk, optical disk, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a general image quality enhancement device based on cyclic consistency loss, which is usually represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0058] The network training module is configured to conduct unsupervised adversarial training through two processes, forward loop and backward loop, and introduce cycle consistency loss to achieve network training using three unpaired real endoscopic images;

[0059] A network parameter update module, which is configured to use a general model training strategy to update the network parameters three times using three types of image quality defect data in one iteration;

[0060] The defect handling module is configured to utilize the forward generator network obtained during the training process to generally handle smoke removal, lighting enhancement, and color correction tasks.

[0061] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A general method for image quality enhancement based on cycle consistency loss, characterized by: It includes the following steps: (1) Through unsupervised adversarial training of the forward and backward cycles, the cycle consistency loss is introduced to achieve network training using three unpaired real endoscopic images; (2) Using a general model training strategy, the network parameters are updated three times using three types of image quality defect data in one iteration; (3) Use the forward generator network obtained during training to universally handle smoke removal, lighting enhancement, and color correction tasks; The network structure of this method includes a generator and a discriminator; The generator consists of an encoder, a converter, and a decoder. The image is resized to 256×256 and then input into the encoder consisting of three convolutional modules; the convolutional module includes a convolutional layer, an instance normalization layer, and a ReLU activation function layer; the convolutional layer in the first convolutional module uses a mirror edge padding of size 3, a stride of 1, and a convolution kernel size of 7×7×64; the convolutional layer edge padding size used in the second convolutional module is 1, a stride of 2, and a convolution kernel size of 3×3×128; the convolutional layer edge padding size used in the third convolutional module is 1, and a stride of 2 , the convolution kernel size is 3×3×256; the decoder consists of two deconvolution modules and one convolution module; the first deconvolution module uses a deconvolution layer with an edge padding size of 1, a stride of 2, and a deconvolution kernel size of 3×3×128; the second deconvolution module uses a deconvolution layer with an edge padding size of 1, a stride of 2, and a deconvolution kernel size of 3×3×64; both deconvolution layers are followed by an instance normalization layer and a ReLU activation function layer; the convolution layer used in the last convolution module is the same as the first convolution module in the encoder, but with 64 input channels and 3 output channels; The output is obtained after instance normalization and Tanh activation function; The converter consists of 9 Resnet Blocks, each of which has the same network structure and uses two convolutional modules combined with skip connections; the convolutional layer used in the first convolutional module uses a mirror edge padding of size 1, a stride of 1, a convolution kernel size of 3×3×256, followed by an instance normalization layer and a ReLU activation function layer, and the second convolutional module uses the same structure but does not include a ReLU activation function layer; The discriminator uses a 70×70 PatchGAN, which consists of five convolutional modules. The convolutional layer parameters used in the five convolutional modules are the same, with a step size of 2, an edge padding size of 1, a convolution kernel size of 4×4, and the number of output channels of 64, 128, 256, 512, and 1, respectively. Instance normalization and Leaky ReLU function activation are performed, where the slope of Leaky ReLU is 0.

2. Instance normalization is not used in the first convolutional module. Given an endoscopic image I with image quality defects defect , input to the generator G ideal Generate a high-quality image I ideal ′, discriminator D ideal By comparing with a high-quality endoscopic image ideal Compare to determine the generated high-quality image I ideal ''s truth or falsity; Generator G defect The purpose is to generate an image with image quality defects I defect ″, discriminator D defect Used to judge I defect Is it a real image with endoscopic image quality defects? defect ; Then for high-quality endoscopic image I ideal Using the same method, first pass the generator G defect Generate an image with quality defects I defect ′, and then through the generator G ideal From I defect ′Restore high-quality endoscopic image I ideal ″, respectively using the discriminator D defect and D ideal To make a judgment; the above process is respectively a forward loop and a backward loop, which are expressed as follows: Maintaining the balance between the generator and the discriminator by applying an adversarial loss function while enhancing images with image quality defects; The least squares loss is used instead of the traditional negative log-likelihood loss function to enhance the stability of training and generate higher image quality results. The forward loop adversarial loss and the backward loop adversarial loss are expressed as follows: The cycle consistency loss is introduced to make the content of the output high-quality endoscopic image consistent with the input image. The cycle consistency loss is expressed as follows: The complete objective loss function is a combination of adversarial loss and cycle consistency loss, expressed as follows: The optimization goal is:

2. The general method for image quality enhancement based on cycle consistency loss according to claim 1, characterized in that: The ReLU activation function is used in the generator, and the tanh function is used to map the result to the range of (-1, 1); the LeakyReLU function with a slope of 0.2 is used in the discriminator.

3. The general method for image quality enhancement based on cycle consistency loss according to claim 2, characterized in that: The Adam optimizer is selected as the optimizer for the image quality enhancement framework.

Images