A colonoscopy image enhancement method based on improved generative adversarial network

Abstract

This invention proposes a colonoscopy image enhancement method based on an improved generative adversarial network, which relates to the field of image data processing. The invention proposes a colonoscopy image data processing process, including acquiring colonoscopy image data, constructing a generator, constructing a discriminator, constructing a colonoscopy image enhancement model, and obtaining colonoscopy-enhanced images. The generator is based on the encoder-decoder structure of ResUNet, combined with a dual-branch attention mechanism, to extract multi-scale features of colonoscopy images. By fusing features at different scales, features in key areas of the image are enhanced to generate enhanced colonoscopy images. The discriminator consists of multiple layers of convolution, combines a specific loss function to evaluate the enhancement effect of the generated image, and guides the generator to produce high-quality colonoscopy-enhanced images through a feedback mechanism.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and in particular relates to a colonoscopy image enhancement method based on an improved generative adversarial network. Background Art

[0002] Colonoscopy generally refers to colonoscopy. Colonoscopy is currently the most direct and accurate method for diagnosing colorectal diseases. Colonoscopy is widely used in clinical practice for the screening of colorectal diseases and the prevention of early colorectal cancer. Colonoscopic images play a vital role in the diagnosis and treatment of diseases. Although colonoscopy technology has made significant progress in the medical field, the capture of colonoscopic images is often affected by multiple factors, such as intestinal movement and insufficient lighting, resulting in poor image quality, especially in low-light and high-noise environments.

[0003] Traditional colonoscopy image enhancement methods improve the visual effect of images by improving their brightness and contrast. However, these traditional methods often lead to over-enhancement or distortion of images under low-light and high-noise conditions, which in turn affects the image quality. In addition, traditional methods are usually unable to effectively deal with color distortion in images, resulting in unnatural color changes in the image during the enhancement process. Therefore, although traditional methods can improve the brightness and contrast of colonoscopy images to a certain extent, these methods are insufficient in low-light, noise and color fidelity, making the image enhancement effect unable to meet the high standards of clinical applications.

[0004] The colonoscopy image enhancement method based on deep learning automatically extracts useful features from the image by learning a large amount of colonoscopy image data, and enhances the image using a deep learning model. However, the colonoscopy image enhancement method based on deep learning still has certain limitations in the enhancement effect, and often has problems such as image blur and loss of details. Secondly, the deep learning-based method has insufficient control over the color and noise of the image, which may lead to color distortion or noise amplification, thereby affecting the image quality and clinical diagnosis effect. Therefore, in response to the limitations of traditional methods and existing deep learning methods in the field of colonoscopy image enhancement, a colonoscopy image enhancement method combining generative adversarial networks and attention mechanisms is proposed. Summary of the Invention

[0005] The present invention provides a colonoscopy image enhancement method based on an improved generative adversarial network, aiming to propose a colonoscopy image enhancement model, wherein the generator is based on the encoder-decoder structure of ResUNet, combined with a dual-branch attention mechanism, to extract multi-scale features of colonoscopy images, and enhances the features of key areas of the image by fusing features of different scales to generate enhanced colonoscopy images; wherein the discriminator consists of multiple layers of convolution, combines a specific loss function to evaluate the enhancement effect of the generated image, and guides the generator to generate high-quality colonoscopy enhanced images through a feedback mechanism.

[0006] The present invention aims to propose a colonoscopy image enhancement model and provide a colonoscopy image enhancement method based on an improved generative adversarial network, which includes the following steps.

[0007] S1. Obtain a colonoscopy image dataset, perform preprocessing on the colonoscopy image dataset, including format conversion, size adjustment, normalization, and denoising operations, and divide the preprocessed colonoscopy image dataset into a training set and a test set.

[0008] S2. Construct a generator by using an improved ResUNet network. The improved ResUNet network is based on the encoder-decoder structure of ResUNet and combines the dual-branch attention mechanism to extract multi-scale features of colonoscopy images and design learnable parameters. Features of different scales are adaptively combined to enhance the features of key areas of the colonoscopy image, and the enhanced colonoscopy image is generated through the decoder.

[0009] S3. Construct a discriminator, which consists of multiple layers of convolution, combines a specific loss function to evaluate the enhancement effect of the generated image, and guides the generator to generate high-quality colonoscopy enhanced images through a feedback mechanism.

[0010] S4. Construct a colonoscopy image enhancement model, which consists of an input, a generator, a discriminator, and an output. It combines a dual-branch attention mechanism based on a generative adversarial network to achieve colonoscopy image enhancement.

[0011] S5. Obtain a colonoscopy enhanced image and input the preprocessed colonoscopy image into the colonoscopy image enhancement model. Through the adversarial game between the generator and the discriminator, the colonoscopy image enhancement model continuously optimizes the image enhancement capability of the generator and the discrimination capability of the discriminator, and finally generates a colonoscopy enhanced image.

[0012] Preferably, the generator in S2 is constructed as follows:

[0013] The encoder extracts the features of the colonoscopy image layer by layer through convolution operations. The dual-branch attention mechanism consists of a channel attention branch and a spatial attention branch. The channel attention branch extracts the important features of the colonoscopy image through weighted pooling and mixed pooling. The multi-layer perceptron is used to optimize the pooled features, generate channel attention weights, and multiply the channel attention weights with the input features element by element to obtain the enhanced colonoscopy image channel features. The spatial attention branch obtains the local and global features of the colonoscopy image through average pooling and maximum pooling, splices the pooled features in the channel dimension, generates spatial attention weights through convolution, and multiplies the spatial attention weights with the input features element by element to obtain the enhanced colonoscopy image spatial features, and designs learnable parameters. Adaptive combination of channel features for enhanced enteroscopic images, , is the input feature map, 、 and are the height, width and number of channels of the input feature map, and is the weight matrix, and is the bias parameter, is the input feature The channel mean of , the global mean of each channel is calculated as , is the input feature The global standard deviation of each channel is calculated as , the input features The channel mean and channel standard deviation are spliced ​​together to obtain the statistical feature vector , , using the weight matrix Statistical eigenvector Perform dimensionality reduction to obtain the features after dimensionality reduction , , using the weight matrix The features after dimensionality reduction Map to scalar space and get parameters , and use the Sigmoid function to set the parameters The value of is limited to the range of [0,1], and finally the learnable parameters are obtained The enhanced colonoscopy features output by the dual-branch attention mechanism are input into the decoder, and the resolution of the feature map is gradually restored through convolution and upsampling operations, and finally the enhanced colonoscopy image is generated.

[0014] Preferably, the generator construction method in S2 combines the dual-branch attention mechanism and the adaptive feature fusion technology to accurately extract and enhance the key feature areas in the colonoscopy image while reducing the interference of irrelevant noise. By designing learnable parameters Adaptive optimization of feature fusion is implemented, enabling the generator to dynamically adjust output features according to the distribution of input data.

[0015] Preferably, in S2, the method for generating the enhanced colonoscopy image is:

[0016] S21. Input colonoscopy image To the encoder, , 、 and are the height, width and number of channels of the colonoscopy image respectively. The encoder consists of three blocks, each of which consists of three convolutions and residual connections. The blocks are connected by maximum pooling. In the first block of the encoder, three 3×3 convolutional layers are used, each with 64 convolution kernels. Each convolution is followed by a ReLU activation function, where the input The number of channels is adjusted to 64 through a 1×1 convolutional layer, and a residual connection is made with the output of the second convolutional layer. The obtained output is used as the input of the third layer, and finally the output features of the first block of the encoder are obtained. , ,feature The features are obtained by downsampling through the maximum pooling operation , ,enter In the second block of the encoder, the structure of the second block of the encoder is the same as that of the first block of the encoder, except that the number of convolution kernels in each layer is 128, and the output features of the second block of the encoder are obtained. , ,feature The features are obtained by downsampling through the maximum pooling operation , ,enter In the third block of the encoder, the structure of the third block of the encoder is the same as that of the first block of the encoder, except that the number of convolution kernels in each layer is 256, and the output features of the third block of the encoder are obtained. , ;

[0017] S22. Input features In the dual-branch attention module, the dual-branch attention module consists of a channel attention branch and a spatial attention branch. For the channel attention branch, the input feature , features are obtained through weighted pooling operation , , The weighted coefficient is the weight assigned to each pixel in the weighted pooling process, and the feature is obtained by the mixed pooling operation. , , It is a trade-off parameter that controls the ratio between weighted pooling and maximum pooling. is a weighted pooling operation, It is the maximum pooling operation, which takes the features and features Input into the multilayer perceptron for feature optimization, , , and is the weight in the multilayer perceptron, ReLU is the activation function, using learnable parameters Features after adaptive combination optimization and Get features , , for features Use the Sigmoid activation function to get the channel attention weight, and combine the channel attention weight with the input feature Multiply element by element to get the enhanced features ; For the spatial attention branch, the input feature , the feature is obtained by average pooling operation , , features are obtained through the maximum pooling operation , , the features are transformed into and features Splicing to get features , features after convolutional layer processing Use the Sigmoid activation function to get the spatial attention weight, and combine the spatial attention weight with the input feature Multiply element by element to get the enhanced features ; Features of the output of the fusion channel attention branch and the features output by the spatial attention branch , and get the output features of the dual-branch attention module , ;

[0018] S23. Input features In the decoder, the structure of the decoder is similar to that of the encoder and consists of three blocks, each of which consists of three convolutions and residual connections. The blocks are connected by transposed convolution upsampling. The features output by each block are spliced ​​with the features output by the corresponding block in the encoder. The number of convolution kernels in each layer of the first block is 256, the number of convolution kernels in each layer of the second block is 128, and the number of convolution kernels in each layer of the third block is 64. Finally, through the 1×1 convolution layer, the enhanced colonoscopy image generated by the generator is obtained. .

[0019] Preferably, in S2, the method for generating enhanced colonoscopy images adopts a dual-branch fusion mechanism of channel attention and spatial attention, which enhances the accuracy of feature extraction and expression through dynamic weight distribution, improves the ability of the colonoscopy image enhancement model to capture key area features, and enables the colonoscopy image enhancement model to effectively distinguish important areas from irrelevant information.

[0020] Preferably, in S3, the method for generating a high-quality colonoscopy enhanced image is:

[0021] Input colonoscopy original image And the enhanced colonoscopy image generated by the generator , , , using the sliding window method to and Split into The image block of size and Input is fed into five layers of convolution, where the size of each convolution kernel is The number of convolution kernels is 64, 128, 256, 512 and 1 respectively. The first four convolution layers are followed by a Leaky ReLU activation function, and the last convolution layer is followed by a Sigmoid activation function. The final output is the score of the image block. The weighted sum of the scores of all image blocks is used to obtain the score of the entire colonoscopy image. Assuming that the original colonoscopy image for , colonoscopy enhanced image generated by the generator for , using the binary cross entropy loss function to minimize the error of the discriminator, for each input sample, the adversarial loss of the discriminator is , , is the evaluation score of the discriminator on the enhanced colonoscopy image; the perceptual loss is introduced to calculate the original colonoscopy image and enhanced enteroscopic images The feature space difference between them, the perceptual loss of the discriminator is , , It is the feature extracted by multi-layer convolution operation. is the L2 norm, which is used to measure the difference in feature space. The colonoscopy image enhancement model is trained in a self-supervised manner. The discriminator uses the structural similarity index to evaluate the similarity of colonoscopy images before and after enhancement. The self-supervised loss of the discriminator is , , The original colonoscopy image is calculated Generating colonoscopy enhanced images The structural similarity between them; the final discriminator loss is the weighted sum of adversarial loss, perceptual loss and self-supervision loss , , 、 and It is the weight of the loss term, which is used to control the contribution of different loss terms to the total loss; the discriminator is based on the loss function Calculate the gradient to update the parameters and repeat the process until the loss of the discriminator converges and the generator is able to generate high-quality colonoscopy enhanced images, and finally obtain high-quality colonoscopy enhanced images.

[0022] Preferably, in said S3, the method for generating high-quality colonoscopy enhanced images, the discriminator combines sliding window evaluation and adversarial, perceptual, and structural similarity losses to achieve high-quality enhancement of colonoscopy images; the generator can effectively extract key features, improve the clarity and contrast of colonoscopy enhanced images, suppress noise and artifacts, and the discriminator finely evaluates the quality of colonoscopy enhanced images, guiding the generator to generate more realistic colonoscopy enhanced images.

[0023] Preferably, the method for constructing the colonoscopy image enhancement model in S4 is:

[0024] The preprocessed colonoscopy image training set is input into the generator to generate enhanced colonoscopy images. The generated colonoscopy enhanced images are input into the discriminator together with the real colonoscopy images. The discriminator evaluates the enhancement effect of the generated colonoscopy enhanced images. The generator and discriminator are continuously optimized through adversarial training. During the training process, the generator and discriminator continuously adjust their respective weights and parameters, and finally reach a balance point by optimizing the loss function. After the training, the generator can generate high-quality colonoscopy enhanced images based on the input colonoscopy images.

[0025] Preferably, the method for constructing the colonoscopy image enhancement model in S4 continuously improves the clarity, contrast and realism of the generated colonoscopy enhanced image through the collaborative optimization of the generator and the discriminator, so that the generator generates high-quality colonoscopy enhanced images. The colonoscopy image enhancement model can not only effectively suppress noise and artifacts, but also optimize the details of the colonoscopy image, making the enhanced colonoscopy image more suitable for medical diagnosis and analysis, thereby providing doctors with more intuitive and reliable auxiliary diagnosis support and improving the efficiency and accuracy of doctors' diagnosis.

[0026] Compared with the prior art, the present invention has the following technical effects:

[0027] The technical solution provided by the present invention proposes a colonoscopy image enhancement model, which combines an improved generative adversarial network and an attention mechanism to generate enhanced colonoscopy images. The generator in the colonoscopy image enhancement model is based on the encoder-decoder structure of ResUNet combined with a dual-branch attention mechanism to extract multi-scale features of the colonoscopy image. By fusing features of different scales, the features of the colonoscopy image can be effectively enhanced. The dual-branch attention module enables the generator to adaptively focus on key areas in the colonoscopy image, especially the lesion area; the discriminator in the colonoscopy image enhancement model extracts features through multi-layer convolution operations, combines a specific loss function to evaluate the quality of the generated image, and introduces a feedback mechanism to guide the generator to generate high-quality colonoscopy enhanced images. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of colonoscopy image data processing provided by the present invention.

[0029] Figure 2 This is a structural diagram of the generator provided by the present invention.

[0030] Figure 3 This is a structural diagram of the generator dual-branch attention module provided by the present invention.

[0031] Figure 4 This is the effect diagram of the colonoscopy image before enhancement provided by the present invention.

[0032] Figure 5 This is the effect diagram of the colonoscopy image enhancement provided by the present invention. DETAILED DESCRIPTION

[0033] The present invention aims to propose a colonoscopy image enhancement method based on an improved generative adversarial network, and proposes a colonoscopy image enhancement model, in which the generator is based on the encoder-decoder structure of ResUNet, combined with a dual-branch attention mechanism, to extract multi-scale features of colonoscopy images, and enhance the features of key areas of the image by fusing features of different scales to generate enhanced colonoscopy images; the discriminator consists of multiple layers of convolution, combines a specific loss function to evaluate the enhancement effect of the generated image, and guides the generator to generate high-quality colonoscopy enhanced images through a feedback mechanism.

[0034] See Figure 1 As shown, a colonoscopy image enhancement method based on an improved generative adversarial network in an embodiment of the present application.

[0035] S1. Obtain a colonoscopy image dataset, perform preprocessing on the colonoscopy image dataset, including format conversion, size adjustment, normalization, and denoising operations, and divide the preprocessed colonoscopy image dataset into a training set and a test set.

[0036] Furthermore, in step S1, for obtaining the colonoscopy image dataset, the colonoscopy image dataset in DICOM format is obtained through the electronic colonoscopy system, the Pydicom library and Pillow library in Python are used to convert the colonoscopy images in DICOM format into colonoscopy images in PNG format, the bilinear interpolation method is used to resize the colonoscopy images to 224×224, the mean variance normalization method is used to normalize the colonoscopy images, the bilateral filtering method is used to remove the noise contained in the colonoscopy images, and the preprocessed colonoscopy image dataset is divided into a training set and a test set in a ratio of 8:2.

[0037] S2. Construct a generator by using an improved ResUNet network. The improved ResUNet network is based on the encoder-decoder structure of ResUNet and combines the dual-branch attention mechanism to extract multi-scale features of colonoscopy images and design learnable parameters. Features of different scales are adaptively combined to enhance the features of key areas of the colonoscopy image, and the enhanced colonoscopy image is generated through the decoder.

[0038] Furthermore, the construction method of the generator in S2 is:

[0039] The structure of the generator is as follows Figure 1As shown, the encoder extracts the features of the colonoscopy image layer by layer through convolution operations. The dual-branch attention mechanism consists of a channel attention branch and a spatial attention branch. The channel attention branch extracts the important features of the colonoscopy image through weighted pooling and mixed pooling. The multi-layer perceptron is used to optimize the pooled features, generate channel attention weights, and multiply the channel attention weights with the input features element by element to obtain the enhanced colonoscopy image channel features. The spatial attention branch obtains the local and global features of the colonoscopy image through average pooling and maximum pooling, splices the pooled features in the channel dimension, generates spatial attention weights through convolution, and multiplies the spatial attention weights with the input features element by element to obtain the enhanced colonoscopy image spatial features, and designs learnable parameters. Adaptive combination of channel features for enhanced enteroscopic images, , is the input feature map, 、 and are the height, width and number of channels of the input feature map, and is the weight matrix, and is the bias parameter, is the input feature The channel mean of , the global mean of each channel is calculated as , is the input feature The global standard deviation of each channel is calculated as , the input features The channel mean and channel standard deviation are spliced ​​together to obtain the statistical feature vector , , using the weight matrix Statistical eigenvector Perform dimensionality reduction to obtain the features after dimensionality reduction , , using the weight matrix The features after dimensionality reduction Map to scalar space and get parameters , and use the Sigmoid function to set the parameters The value of is limited to the range of [0,1], and finally the learnable parameters are obtained The enhanced colonoscopy features output by the dual-branch attention mechanism are input into the decoder, and the resolution of the feature map is gradually restored through convolution and upsampling operations, and finally the enhanced colonoscopy image is generated.

[0040] Furthermore, in S2, the method for generating the enhanced colonoscopy image is:

[0041] S21. Input colonoscopy image To the encoder, , 224, 224 and 3 are the height, width and number of channels of the colonoscopy image respectively. The encoder consists of three blocks, each of which consists of three convolutions and residual connections. The blocks are connected by maximum pooling. The input In the first block of the encoder, three 3×3 convolutional layers are used, each with 64 convolution kernels. Each convolution is followed by a ReLU activation function, where the input The number of channels is adjusted to 64 through a 1×1 convolutional layer, and a residual connection is made with the output of the second convolutional layer. The obtained output is used as the input of the third layer, and finally the output features of the first block of the encoder are obtained. , ,feature The features are obtained by downsampling through the maximum pooling operation , ,enter In the second block of the encoder, the structure of the second block of the encoder is the same as that of the first block of the encoder, except that the number of convolution kernels in each layer is 128, and the output features of the second block of the encoder are obtained. , ,feature The features are obtained by downsampling through the maximum pooling operation , ,enter In the third block of the encoder, the structure of the third block of the encoder is the same as that of the first block of the encoder, except that the number of convolution kernels in each layer is 256, and the output features of the third block of the encoder are obtained. , ;

[0042] S22. Input features To the dual-branch attention module, its structure is as follows Figure 3 As shown, the dual-branch attention module consists of a channel attention branch and a spatial attention branch. For the channel attention branch, the input feature , features are obtained through weighted pooling operation , , The weighted coefficient is the weight assigned to each pixel in the weighted pooling process, and the feature is obtained by the mixed pooling operation. , , It is a trade-off parameter that controls the ratio between weighted pooling and maximum pooling. is a weighted pooling operation, It is the maximum pooling operation, which takes the features and features Input into the multilayer perceptron for feature optimization, , , and is the weight in the multilayer perceptron, ReLU is the activation function, using learnable parameters Features after adaptive combination optimization and Get features , , for features Use the Sigmoid activation function to get the channel attention weight, and combine the channel attention weight with the input feature Multiply element by element to get the enhanced features ; For the spatial attention branch, the input feature , the feature is obtained by average pooling operation , , features are obtained through the maximum pooling operation , , the features are transformed into and features Splicing to get features , features after convolutional layer processing Use the Sigmoid activation function to get the spatial attention weight, and combine the spatial attention weight with the input feature Multiply element by element to get the enhanced features ; Features of the output of the fusion channel attention branch and the features output by the spatial attention branch , and get the output features of the dual-branch attention module , ;

[0043] S23. Input features In the decoder, the decoder contains three blocks, each of which consists of three convolutions and residual connections. The blocks are connected by transposed convolution upsampling. The features output by each block are spliced ​​with the features output by the corresponding block in the encoder. The number of convolution kernels in each layer of the first block is 256, the number of convolution kernels in each layer of the second block is 128, and the number of convolution kernels in each layer of the third block is 64. Finally, through the 1×1 convolution layer, the enhanced colonoscopy image generated by the generator is obtained. .

[0044] S3. Construct a discriminator, which consists of multiple layers of convolution, combines a specific loss function to evaluate the enhancement effect of the generated image, and guides the generator to generate high-quality colonoscopy enhanced images through a feedback mechanism.

[0045] Furthermore, in S3, the method for generating a high-quality colonoscopy enhanced image is:

[0046] Input colonoscopy original image And the enhanced colonoscopy image generated by the generator , , , the image is transformed into and Divide the image into 4×4 blocks and and The input is sent to five layers of convolution, where the size of each convolution kernel is 4×4, and the number of convolution kernels is 64, 128, 256, 512 and 1 respectively. The first four convolution layers are followed by a Leaky ReLU activation function, and the last convolution layer is followed by a Sigmoid activation function. Finally, the score of the image block is output. The weighted sum of the scores of all image blocks is used to obtain the score of the entire colonoscopy image. Assuming that the original colonoscopy image for , colonoscopy enhanced image generated by the generator for , using the binary cross entropy loss function to minimize the error of the discriminator, for each input sample, the adversarial loss of the discriminator is , , is the evaluation score of the discriminator on the enhanced colonoscopy image; the perceptual loss is introduced to calculate the original colonoscopy image and enhanced enteroscopic images The feature space difference between them, the perceptual loss of the discriminator is , , It is the feature extracted by multi-layer convolution operation. is the L2 norm, which is used to measure the difference in feature space. The colonoscopy image enhancement model is trained in a self-supervised manner. The discriminator uses the structural similarity index to evaluate the similarity of colonoscopy images before and after enhancement. The self-supervised loss of the discriminator is , , The original colonoscopy image is calculated Generating colonoscopy enhanced images The structural similarity between them; the final discriminator loss is the weighted sum of adversarial loss, perceptual loss and self-supervision loss , , 、 and It is the weight of the loss term, which is used to control the contribution of different loss terms to the total loss; the discriminator is based on the loss function Calculate the gradient to update the parameters and repeat the process until the loss of the discriminator converges and the generator is able to generate high-quality colonoscopy enhanced images, and finally obtain the colonoscopy enhanced image.

[0047] S4. Construct a colonoscopy image enhancement model, which consists of an input, a generator, a discriminator, and an output. It combines a dual-branch attention mechanism based on a generative adversarial network to achieve colonoscopy image enhancement.

[0048] Furthermore, the method for constructing the colonoscopy image enhancement model in S4 is as follows:

[0049] The preprocessed colonoscopy image training set is input into the generator to generate enhanced colonoscopy images. The generated colonoscopy enhanced images are input into the discriminator together with the real colonoscopy images. The discriminator evaluates the enhancement effect of the generated colonoscopy enhanced images. The generator and discriminator are continuously optimized through adversarial training. During the training process, the generator and discriminator continuously adjust their respective weights and parameters, and finally reach a balance point by optimizing the loss function. After the training, the generator can generate high-quality colonoscopy enhanced images based on the input colonoscopy images.

[0050] Furthermore, in step S4, the colonoscopy image enhancement model was implemented based on the Python language using the Pytorch framework, the optimizer used Adam, the generator in the improved generative adversarial network used a binary cross-entropy loss function, and the discriminator used the weighted sum of adversarial loss, perceptual loss, and feature consistency loss as the total loss of the discriminator. The learning rate was set to 0.0002, the batch size was 64, the training rounds were 50, and indicators such as peak signal-to-noise ratio and structural similarity index were used to evaluate the performance of the colonoscopy image enhancement model.

[0051] S5. Obtain a colonoscopy enhanced image and input the preprocessed colonoscopy image into the colonoscopy image enhancement model. Through the adversarial game between the generator and the discriminator, the colonoscopy image enhancement model continuously optimizes the image enhancement capability of the generator and the discrimination capability of the discriminator, and finally generates a colonoscopy enhanced image.

[0052] Furthermore, in step S5, for obtaining colonoscopy enhanced images, the colonoscopy images preprocessed in step S1 are input into the colonoscopy image enhancement model, such as Figure 4 As shown, Figure 4 The image of the colonoscopy image before enhancement is shown. The enhanced colonoscopy image is generated by the generator, the enhancement effect of the generated image is evaluated by the discriminator, and the feedback mechanism is used to guide the generator to generate high-quality colonoscopy enhanced images. Figure 5 As shown, Figure 5 The image shows the enhanced effect of the colonoscopy image after being processed by the colonoscopy image enhancement model.

[0053] The above are only preferred embodiments of the present invention. It should be pointed out that those skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A colonoscopy image enhancement method based on an improved generative adversarial network, characterized in that: The following steps are involved: S1. Obtain a colonoscopy image dataset, perform preprocessing on the colonoscopy image dataset, including format conversion, size adjustment, normalization, and denoising operations, and divide the preprocessed colonoscopy image dataset into a training set and a test set; S2. Construct a generator by using an improved ResUNet network. The improved ResUNet network is based on the encoder-decoder structure of ResUNet and combines the dual-branch attention mechanism to extract multi-scale features of colonoscopy images and design learnable parameters. Adaptively combine features of different scales to enhance the features of key areas of the colonoscopy image, and generate the enhanced colonoscopy image through the decoder. The learnable parameters The construction method is: , is the input feature map, 、 and are the height, width and number of channels of the input feature map, and is the weight matrix, and is the bias parameter, is the input feature The channel mean of , the global mean of each channel is calculated as , is the input feature The global standard deviation of each channel is calculated as ; S3. Construct a discriminator, which consists of multiple layers of convolution, combines a specific loss function to evaluate the enhancement effect of the generated image, and guides the generator to generate high-quality colonoscopy enhanced images through a feedback mechanism; S4. Construct a colonoscopy image enhancement model, which consists of an input, a generator, a discriminator, and an output. It combines a dual-branch attention mechanism based on a generative adversarial network to enhance colonoscopy images. S5. Obtain a colonoscopy enhanced image and input the preprocessed colonoscopy image into the colonoscopy image enhancement model. Through the adversarial game between the generator and the discriminator, the colonoscopy image enhancement model continuously optimizes the image enhancement capability of the generator and the discrimination capability of the discriminator, and finally generates a colonoscopy enhanced image.

2. The colonoscopy image enhancement method based on an improved generative adversarial network according to claim 1, characterized in that: The construction method of the generator in S2 is: The encoder extracts the features of the colonoscopy image layer by layer through convolution operations. The dual-branch attention mechanism consists of a channel attention branch and a spatial attention branch. The channel attention branch extracts the important features of the colonoscopy image through weighted pooling and mixed pooling. The multi-layer perceptron is used to optimize the pooled features, generate channel attention weights, and multiply the channel attention weights with the input features element by element to obtain the enhanced colonoscopy image channel features. The spatial attention branch obtains the local and global features of the colonoscopy image through average pooling and maximum pooling, splices the pooled features in the channel dimension, generates spatial attention weights through convolution, and multiplies the spatial attention weights with the input features element by element to obtain the enhanced colonoscopy image spatial features, and designs learnable parameters. Adaptive combination enhances the channel features of the colonoscopy image, and the input features The channel mean and channel standard deviation are spliced ​​together to obtain the statistical feature vector , , using the weight matrix Statistical eigenvector Perform dimensionality reduction to obtain the features after dimensionality reduction , , using the weight matrix The features after dimensionality reduction Map to scalar space and get parameters , and use the Sigmoid function to set the parameters The value of is limited to the range of [0,1], and finally the learnable parameters are obtained The enhanced colonoscopy features output by the dual-branch attention mechanism are input into the decoder, and the resolution of the feature map is gradually restored through convolution and upsampling operations, and finally the enhanced colonoscopy image is generated.

3. The colonoscopy image enhancement method based on improved generative adversarial network according to claim 2, characterized in that: The method for generating enhanced colonoscopy images is: S21. Input colonoscopy image To the encoder, , 、 and are the height, width and number of channels of the colonoscopy image respectively. The encoder consists of three blocks, each of which consists of three convolutions and residual connections. The blocks are connected by maximum pooling. In the first block of the encoder, three 3×3 convolutional layers are used, each with 64 convolution kernels. Each convolution is followed by a ReLU activation function, where the input The number of channels is adjusted to 64 through a 1×1 convolutional layer, and a residual connection is made with the output of the second convolutional layer. The obtained output is used as the input of the third layer, and finally the output features of the first block of the encoder are obtained. , ,feature The features are obtained by downsampling through the maximum pooling operation , ,enter In the second block of the encoder, the structure of the second block of the encoder is the same as that of the first block of the encoder, except that the number of convolution kernels in each layer is 128, and the output features of the second block of the encoder are obtained. , ,feature The features are obtained by downsampling through the maximum pooling operation , ,enter In the third block of the encoder, the structure of the third block of the encoder is the same as that of the first block of the encoder, except that the number of convolution kernels in each layer is 256, and the output features of the third block of the encoder are obtained. , ; S22. Input features In the dual-branch attention module, the dual-branch attention module consists of a channel attention branch and a spatial attention branch. For the channel attention branch, the input feature , features are obtained through weighted pooling operation , , The weighted coefficient is the weight assigned to each pixel in the weighted pooling process, and the feature is obtained by the mixed pooling operation. , , It is a trade-off parameter that controls the ratio between weighted pooling and maximum pooling. is a weighted pooling operation, It is the maximum pooling operation, which takes the features and features Input into the multilayer perceptron for feature optimization, , , and is the weight in the multilayer perceptron, ReLU is the activation function, using learnable parameters Features after adaptive combination optimization and Get features , , for features Use the Sigmoid activation function to get the channel attention weight, and combine the channel attention weight with the input feature Multiply element by element to get the enhanced features ; For the spatial attention branch, the input features , the feature is obtained by average pooling operation , , features are obtained through the maximum pooling operation , , the features are transformed into and features Splicing to get features , features after convolutional layer processing Use the Sigmoid activation function to get the spatial attention weight, and combine the spatial attention weight with the input feature Multiply element by element to get the enhanced features ; Fusion channel attention branch output features and the features output by the spatial attention branch , and get the output features of the dual-branch attention module , ; S23. Input features In the decoder, the decoder contains three blocks, each of which consists of three convolutions and residual connections. The blocks are connected by transposed convolution upsampling. The features output by each block are spliced ​​with the features output by the corresponding block in the encoder. The number of convolution kernels in each layer of the first block is 256, the number of convolution kernels in each layer of the second block is 128, and the number of convolution kernels in each layer of the third block is 64. Finally, through the 1×1 convolution layer, the enhanced colonoscopy image generated by the generator is obtained. .

4. The colonoscopy image enhancement method based on an improved generative adversarial network according to claim 3, characterized in that: In S3, the method for generating a high-quality colonoscopy enhanced image is: Input colonoscopy original image And the enhanced colonoscopy image generated by the generator , , , using the sliding window method to and Split into The image block of size and Input is fed into five layers of convolution, where the size of each convolution kernel is The number of convolution kernels is 64, 128, 256, 512 and 1 respectively. The first four convolution layers are followed by a Leaky ReLU activation function, and the last convolution layer is followed by a Sigmoid activation function. The final output is the score of the image block. The weighted sum of the scores of all image blocks is used to obtain the score of the entire colonoscopy image. Assuming that the original colonoscopy image for , colonoscopy enhanced image generated by the generator for , using the binary cross entropy loss function to minimize the error of the discriminator, for each input sample, the adversarial loss of the discriminator is , , is the evaluation score of the discriminator on the enhanced colonoscopy image; the perceptual loss is introduced to calculate the original colonoscopy image and enhanced enteroscopic images The feature space difference between them, the perceptual loss of the discriminator is , , It is the feature extracted by multi-layer convolution operation. is the L2 norm, which is used to measure the difference in feature space. The colonoscopy image enhancement model is trained in a self-supervised manner. The discriminator uses the structural similarity index to evaluate the similarity of colonoscopy images before and after enhancement. The self-supervised loss of the discriminator is , , The original colonoscopy image is calculated Generating colonoscopy enhanced images The structural similarity between them; the final discriminator loss is the weighted sum of adversarial loss, perceptual loss and self-supervision loss , , 、 and is the weight of the loss term, which is used to control the contribution of different loss terms to the total loss; The discriminator is based on the loss function Calculate the gradient to update the parameters and repeat the process until the loss of the discriminator converges and the generator is able to generate high-quality colonoscopy enhanced images, and finally obtain high-quality colonoscopy enhanced images.

5. The colonoscopy image enhancement method based on improved generative adversarial network according to claim 4, characterized in that: The method for constructing the colonoscopy image enhancement model in S4 is as follows: The preprocessed colonoscopy image training set is input into the generator to generate enhanced colonoscopy images. The generated colonoscopy enhanced images and the real colonoscopy images are input into the discriminator together. The discriminator evaluates the enhancement effect of the generated colonoscopy enhanced images. The generator and discriminator are continuously optimized through adversarial training. During the training process, the generator and discriminator continuously adjust their respective weights and parameters, and eventually reach a balance point by optimizing the loss function. After the training, the generator can generate high-quality colonoscopy-enhanced images based on the input colonoscopy images.

Images