Low-illumination digestive endoscope image enhancement method, device and equipment

Abstract

The invention relates to the technical field of image processing, and provides a low-illumination digestive endoscope image enhancement method, device and equipment, and the method comprises the steps: carrying out the normalization processing of the pixel value of each pixel point in an initial target image, and obtaining a target image; performing feature extraction on the target image to identify brightness features of each pixel point in the target image to obtain an enhanced description matrix; respectively using a convolutional layer for the target image and the enhanced description matrix to obtain feature vectors, carrying out feature fusion on the feature vectors to obtain a fused image feature vector, and carrying out feature extraction and denoising operation on the fused image feature vector to obtain a denoised image feature vector; performing image conversion on the de-noised image feature vector based on the enhanced description matrix to obtain an enhanced image feature vector; and performing a de-normalization operation on the feature value of each pixel point of the feature vector of the enhanced image to obtain a target enhanced image. According to the invention, self-supervised learning and useful feature extraction can be realized, and image brightness and image quality can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a low-light digestive endoscope image enhancement method, device and equipment. Background Art

[0002] In the technical field of image processing and recognition, since the image acquisition process is limited by the influence of environmental structure and ambient brightness, some images in low-light conditions are often acquired. These low-light images often affect subsequent image recognition and cause image recognition errors.

[0003] For example, the human body is completely dark inside, and a cold light source is needed to illuminate the gastrointestinal tract during a gastrointestinal endoscopy. Different parts of the digestive tract are examined in sequence by the order in which the light is shone. However, due to the complexity of the digestive tract structure, technical limitations of the cold light source, inappropriate configuration of endoscope parameters, etc., the brightness of the digestive endoscopy image is often too dark, which greatly affects the inspection effect and the diagnosis of digestive tract diseases.

[0004] At present, the traditional image enhancement method based on supervised learning uses image pairs composed of low-light images and high-light images to construct a training set for training AI models. This method has the following defects: 1) The training set has very high requirements for image pairs. The content of the low-light images and high-light images used in the training set must be completely consistent and the positions of the pixels must correspond completely. However, this type of digestive endoscope image pairs are difficult to collect, so the supervised learning solution is difficult to implement in the actual implementation process; 2) This type of model cannot automatically distinguish whether the image brightness is too low, and will indiscriminately process images with good brightness. Resulting in excessively high image brightness; 3) This type of model only processes the image brightness, but cannot process the noise in the image. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to overcome or partially overcome the above problems by providing a low-light digestive endoscope image enhancement method, device and equipment.

[0006] One aspect of the present invention provides a method for enhancing a low-light digestive endoscope image, the method comprising: Normalizing the pixel value of each pixel in the initial target image to be enhanced to obtain the target image; Based on a preset image enhancement network, feature extraction is performed on the target image to identify the brightness features of each pixel in the target image, and an enhancement description matrix is ​​obtained, wherein the enhancement description matrix is ​​a two-dimensional matrix with the same scale as the target image; Using convolutional layers to obtain feature vectors for the target image and the enhanced description matrix respectively, fusing the feature vectors to obtain a fused image feature vector, performing feature extraction on the fused image feature vector, and performing a denoising operation on the extracted feature values ​​to obtain a denoised image feature vector; Performing image transformation on the denoised image feature vector based on the enhanced description matrix, so as to enhance the feature value of the denoised image feature vector at each pixel point to different degrees according to the brightness characteristics of the corresponding pixel point, thereby obtaining an enhanced image feature vector; A denormalization operation is performed on the eigenvalues ​​of each pixel point of the enhanced image eigenvector to obtain a target enhanced image of the initial target image.

[0007] Furthermore, the image conversion of the denoised image feature vector based on the enhanced description matrix includes: The preset image conversion model is used to realize the image conversion of the denoised image feature vector based on the enhanced description matrix. The image conversion model is expressed as: Among them, p refers to the pixel point in the image, Φ L (p) refers to the eigenvalue of the enhanced description matrix at pixel p, L D (p) refers to the eigenvalue of the denoised image feature vector at pixel p, L E (p) refers to the calculated eigenvalue of the enhanced image feature vector at pixel point p.

[0008] Furthermore, the image enhancement network is used to complete the following operations: Iteratively extracting features of the target image through a plurality of feature extraction modules connected in sequence, and obtaining a feature matrix of a plurality of channels for representing brightness features; A convolution operation is performed on the feature matrices of multiple channels to obtain a two-dimensional matrix of one channel, and the obtained two-dimensional matrix is ​​used as an enhanced description matrix.

[0009] Furthermore, before extracting features of the target image based on a preset image enhancement network to identify brightness features of each pixel in the target image, the method further includes: The image enhancement network is trained based on a preset total loss function of the image enhancement network; the total loss function of the image enhancement network includes a self-supervised loss function of a preset first weight coefficient, and the self-supervised loss function is expressed as: Among them, 0<first weight coefficient≤1, Loss s is the self-supervised loss, L refers to low-light images, and F M refers to the image enhancement network, F M(L) is the output result of the low-light image after being processed by the image enhancement network, ω is a custom parameter for self-supervised learning, 0<ω<1, Lω is the calculation result after optimizing the normalized pixel value of each pixel in L using the parameter ω, and F M (Lω) is the result of Lω being processed by the image enhancement network. The calculation formula of Lω is as follows: Among them, L(p) is the normalized pixel value of a single pixel in the low-light image.

[0010] Furthermore, the total loss function of the image enhancement network also includes a low-light image enhancement loss function of a preset second weight coefficient, and the low-light image enhancement loss function is expressed as: Among them, 0<second weight coefficient≤1, Loss C is the low-light image enhancement loss, L M = F M (L), F M (L M ) is the result of the second processing of the low-light image by the image enhancement network after the image enhancement network has processed the output result. E is a matrix with the same scale as the enhancement description matrix and all elements are 1.

[0011] Furthermore, the total loss function of the image enhancement network also includes a highlight image enhancement loss function of a preset third weight coefficient, and the highlight image enhancement loss function is expressed as: Among them, 0<third weight coefficient≤1, Loss H is the highlight image enhancement loss, H is the highlight image, and F M (H) is the output result of the highlight image after being processed by the image enhancement network.

[0012] Further, the target image and the enhanced description matrix are respectively subjected to convolutional layers to obtain feature vectors, the feature vectors are subjected to feature fusion to obtain a fused image feature vector, the fused image feature vector is subjected to feature extraction, and a denoising operation is performed on the extracted feature values ​​to obtain a denoised image feature vector, including: Based on the preset image denoising network, the following operations are performed on the target image and the enhanced description matrix input into the image denoising network: Performing a convolution operation on the target image to obtain a first convolution input image feature vector; Performing a convolution operation on the enhanced description matrix to obtain a second convolution enhanced description feature vector; Performing a concatenation operation on the first convolution input image feature vector and the second convolution enhancement description feature vector to obtain a fused image feature vector of the target image and the enhancement description matrix; At least one feature extraction module and at least one soft threshold denoising module respectively perform feature extraction operations and soft threshold denoising operations on the fused image feature vector to obtain an initial denoised image feature vector; wherein the soft threshold denoising module is used to perform the following operations: Wherein, v is the characteristic value input to the soft threshold denoising module, δ is the preset soft threshold parameter, when v is greater than 0, Sign(v) = 1, when v is less than 0, Sign(v) = -1; A convolution operation is performed on the initial denoised image feature vector to obtain at least one feature matrix having the same number of channels and scale as the target image, and the obtained at least one feature matrix is ​​used as the denoised image feature vector.

[0013] Furthermore, before using convolution layers to calculate the target image and the enhanced description matrix to form feature vectors, the method further includes: Performing model training on the image denoising network based on a preset total loss function of the image denoising network; The total loss function of the image denoising network is obtained by weighted addition of the low-light image denoising loss and the high-light image denoising loss; wherein, The low-light image denoising loss function is expressed as: Among them, Loss D (L) refers to the low-light image denoising loss, F D (L) is the output result after the low-light image is processed by the image denoising network, L refers to the low-light image; The high light image denoising loss is expressed as: Among them, Loss D (H) refers to the denoising loss of high-light images, F D (H) is the output result after the highlight image is processed by the image denoising network, and H is the highlight image.

[0014] Furthermore, the feature extraction module is used to complete the following operations: Performing iterative convolution feature extraction on the image features of the input feature extraction module through multiple feature extraction convolution layers; In the process of convolutional feature extraction, the weight of channel features is adaptively adjusted through the preset channel attention module to increase the weight of the channel used to characterize brightness; The image features obtained after performing convolution feature extraction and the image features input to the feature extraction module are element-wise added to obtain the output features of the feature extraction module.

[0015] Another aspect of the present invention provides a low-light digestive endoscope image enhancement device, the device comprising: An image processing module is used to normalize the pixel value of each pixel in the initial target image to be enhanced to obtain a target image; An image enhancement module is used to extract features of a target image based on a preset image enhancement network to identify brightness features of each pixel in the target image, and obtain an enhancement description matrix, wherein the enhancement description matrix is ​​a two-dimensional matrix with the same scale as the target image; An image denoising module, used to obtain feature vectors using convolution layers for the target image and the enhanced description matrix respectively, perform feature fusion on the feature vectors to obtain a fused image feature vector, perform feature extraction on the fused image feature vector, and perform denoising operation on the extracted feature values ​​to obtain a denoised image feature vector; An image conversion module is used to perform image conversion on the denoised image feature vector based on the enhancement description matrix, so as to enhance the feature value of the denoised image feature vector at each pixel point to different degrees according to the brightness characteristics of the corresponding pixel point, thereby obtaining an enhanced image feature vector; The image processing module is further used to perform a denormalization operation on the eigenvalues ​​of each pixel point of the enhanced image eigenvector to obtain a target enhanced image of the initial target image.

[0016] In another aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the low-light digestive endoscopy image enhancement method described in the above items is implemented.

[0017] The present invention provides a low-light digestive endoscope image enhancement method, device and equipment, which normalize the pixel values ​​of each pixel in an initial target image to be enhanced to obtain a target image, and extract features from the target image to identify the brightness features of each pixel in the target image to obtain an enhanced description matrix; at the same time, convolution layers are used to obtain feature vectors for the enhanced description matrix and the target image respectively, and image fusion is performed on the feature vectors to obtain a fused image feature vector, and feature extraction is performed on the fused image feature vector, and a denoising operation is performed on the extracted feature values ​​to obtain a denoised image feature vector; finally, based on the enhanced description matrix, image conversion is performed on the denoised image feature vector to obtain an enhanced image feature vector. Since the enhanced description matrix can reflect the brightness features of each pixel, when the denoised image is converted, the feature values ​​of the denoised image feature vector of the pixel with lower brightness can be highly enhanced, thereby achieving the purpose of improving image quality and image enhancement for low-light images at the same time.

[0018] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. In the accompanying drawings: Figure 1 A flowchart of a low-light digestive endoscope image enhancement method according to an embodiment of the present invention; Figure 2 A network architecture diagram of a low-light digestive endoscopy image enhancement model according to an embodiment of the present invention; Figure 3 A comparison diagram of an enhanced image obtained by using the low-light digestive endoscope image enhancement method of the present invention; wherein, (a) is a low-light endoscopic image of the stomach; (b) is the enhanced gastric endoscopic image; (c) is a low-light endoscopic image of the intestine; (d) is the enhanced endoscopic image of the colon; Figure 4 Schematic diagram of the structure of a low-light digestive endoscope image enhancement device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0021] Those skilled in the art will understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0022] Embodiment 1 The embodiment of the present invention provides a method for enhancing a low-light digestive endoscope image. Figure 1 As shown, the low-light image enhancement method comprises the following steps: S1, normalizing the pixel value of each pixel in the initial target image to be enhanced to obtain a target image; It should be noted that, in the embodiment of the present invention, the actual pixel value of each pixel of the initial target image is normalized by dividing the pixel value of each pixel of the initial target image by 255, so that the normalized pixel value of each pixel of the target image is a floating point number greater than 0 and less than 1. Figure 2 The specific embodiment uses RGB image as the network model built as the specific embodiment. At this time, the initial target image has three channels, that is, each pixel has pixel values ​​of three colors. When performing normalization operation, it is necessary to perform normalization operation on the three pixel values ​​of each pixel. It can be understood that depending on the application scenario, the initial target image used is also different. The present invention is also applicable to the enhancement of other target images with different numbers of channels.

[0023] S2. Extracting features of the target image based on a preset image enhancement network to identify brightness features of each pixel in the target image, and obtaining an enhanced description matrix, wherein the enhanced description matrix is ​​a two-dimensional matrix with the same scale as the target image; In a specific embodiment of the present invention, the brightness features of each pixel in the target image can be identified by identifying low-light pixels as values ​​greater than 0 and less than 1. After a low-light pixel is processed by the image enhancement network, the eigenvalue of the enhancement description matrix at the corresponding pixel is identified as 0.75, and the eigenvalue of a high-light (normal-light) pixel is identified as 1.

[0024] S3, using convolution layers to obtain feature vectors for the target image and the enhanced description matrix respectively, performing feature fusion on the feature vectors to obtain a fused image feature vector, performing feature extraction on the fused image feature vector, and performing a denoising operation on the extracted feature values ​​to obtain a denoised image feature vector; In a specific embodiment of the present invention, the cocat module is used to perform feature fusion on the feature vector obtained after performing the convolution operation on the target image and the enhanced description matrix, so as to further improve the denoising effect. Soft threshold denoising is performed on the feature values ​​in the network parameters, and the amplitude of the signal is compared. When the amplitude of the signal is less than the set threshold, the signal is set to zero; when the amplitude of the signal is greater than or equal to the threshold, the original signal is retained, which can effectively suppress the noise in the signal while retaining the main features of the signal.

[0025] S4, performing image conversion on the denoised image feature vector based on the enhanced description matrix, so as to enhance the feature value of the denoised image feature vector at each pixel point to different degrees according to the brightness characteristics of the corresponding pixel point, thereby obtaining an enhanced image feature vector; In an embodiment of the present invention, the image conversion of the denoised image feature vector based on the enhanced description matrix is ​​mainly to enhance the eigenvalues ​​of the denoised image feature vector at low-light pixels to different degrees according to the degree of low light, and the eigenvalues ​​of high-light (normal light) pixels do not need to be enhanced again, thereby obtaining an enhanced image feature vector in which each pixel is high light. It should be noted here that in a specific embodiment of the present invention, for an RGB image, its denoised image feature vector has eigenvalues ​​of three different channels at each pixel, and the eigenvalues ​​in each channel need to be image converted by the eigenvalue of the enhanced description matrix at the same pixel, so that the eigenvalues ​​of the three different channels of the noisy image feature vector at each pixel are enhanced.

[0026] S5. Perform a denormalization operation on the eigenvalues ​​of each pixel point of the enhanced image eigenvector to obtain a target enhanced image of the initial target image.

[0027] In the embodiment of the present invention, a denormalization operation needs to be performed on the enhanced image feature vector, that is, the eigenvalues ​​of each pixel point of the enhanced image feature vector are multiplied by 255 to obtain a real image. It should be noted here that for RGB images, the enhanced image feature vector has a feature matrix of three different channels at each pixel point, and the eigenvalues ​​in each channel need to be denormalized.

[0028] The present invention can enhance low-light images while avoiding re-enhancement of high-light images, thereby effectively ensuring image quality. At the same time, the present invention also processes noise in the image, thereby further improving image quality.

[0029] Figure 2 The network architecture diagram of a low-light digestive endoscope image enhancement model according to an embodiment of the present invention is shown. M Obtain the enhanced description matrix of the target image and use the image denoising network F D The target image is denoised and the image conversion model is used to perform image conversion.

[0030] Furthermore, the image enhancement network of the embodiment of the present invention is also referred to as F M Network, through F M The enhanced description matrix output by the network is represented as Φ L : (1) The enhancement description matrix can be enhanced to different degrees according to different areas of the image. The value of the enhancement description matrix in the low-light area is non-1, while the value of the enhancement description matrix in the highlight area is 1. If the size of an input image is 512*512, the size of the generated enhancement description matrix is ​​also 512*512.

[0031] Specifically, the image enhancement network in the embodiment of the present invention is used to complete the following operations: perform iterative feature extraction on the target image through multiple feature extraction modules connected in sequence to obtain a feature matrix of multiple channels for representing brightness features; perform a convolution operation on the feature matrices of multiple channels to obtain a two-dimensional matrix of one channel, and use the obtained two-dimensional matrix as an enhanced description matrix.

[0032] Furthermore, the feature extraction module in the embodiment of the present invention is used to complete the following operations: performing iterative convolution feature extraction on the image features of the input feature extraction module through multiple feature extraction convolution layers; in the process of convolution feature extraction, adaptively adjusting the weight of the channel features through a preset channel attention module to increase the weight of the channel used to characterize brightness; performing element-wise addition operation on the image features obtained after performing the convolution feature extraction and the image features of the input feature extraction module to obtain the output features of the feature extraction module.

[0033] Specifically, refer to Figure 2 In the specific embodiment of the present invention, the image enhancement network F MIt consists of three feature extraction modules and one 3*3 convolutional layer. The three feature extraction modules have exactly the same structure, including multiple convolutional layers with different numbers of channels and one ECA (Efficient Channel Attention Module) channel attention module. Each convolutional layer of the feature extraction module uses the ReLU activation function. The ECA channel attention module adaptively adjusts the weights of channel features, so that the network can better focus on important features and suppress unimportant features.

[0034] Specifically, in each feature extraction module, the first convolution layer and the second convolution layer have 32 channels, that is, 32 3*3 convolution kernels are used to extract the input features, and the thickness of each convolution kernel is the same as the number of channels input to the convolution kernel, and the ReLU activation function is used for nonlinear transformation, and finally 32 different feature matrices are output; the output features of the second convolution layer are input to the ECA (Efficient Channel Attention Module) channel attention module, at this time, the ECA channel attention module adaptively adjusts the weight of the channel features, increases the weight of the channel that can better reflect the brightness, and reduces the weight of the channel that is not sensitive to brightness; the third convolution layer uses 4 1*1 convolution kernels to extract the input features, and the thickness of each convolution kernel is the same as the number of channels input to the convolution kernel, forming 4 different feature matrices; the fourth convolution layer uses 32 1*1 convolution kernels to extract the input features, forming 32 different feature matrices; finally, the output features of the fourth convolution kernel and the features of the input feature extraction module are element-wise added to obtain the output features of the feature extraction module. The feature extraction modules all use a residual network structure to prevent the gradient of the multi-layer network from disappearing.

[0035] Furthermore, the image enhancement network F of the present invention M Three feature extraction modules are used to perform iterative feature extraction, and a 3*3 convolutional layer is used to reduce the output features of the third feature extraction module to 1 channel, so that the final feature matrix is ​​an enhanced description matrix with the same size as the original image and 1 channel. For example, if the original image is 512*512*3, the output enhanced description matrix is ​​512*512*1.

[0036] Furthermore, in step S3, the embodiment of the present invention uses convolution layers to obtain feature vectors for the target image and the enhancement description matrix respectively, performs feature fusion on the feature vectors to obtain a fused image feature vector, performs feature extraction on the fused image feature vector, and performs denoising operation on the extracted eigenvalues ​​to obtain a denoised image feature vector, including: performing the following operations on the target image and the enhancement description matrix input into the image denoising network based on a preset image denoising network: performing a convolution operation on the target image to obtain a first convolution input image feature vector; performing a convolution operation on the enhancement description matrix to obtain a second convolution enhancement description feature vector; performing a splicing operation on the first convolution input image feature vector and the second convolution enhancement description feature vector to obtain a fused image feature vector of the target image and the enhancement description matrix; performing a feature extraction operation and a soft threshold denoising operation on the fused image feature vector respectively through at least one feature extraction module and at least one soft threshold denoising module to obtain an initial denoised image feature vector; performing a convolution operation on the initial denoised image feature vector to obtain at least one feature matrix having the same number of channels and scale as the target image, and using the obtained at least one feature matrix as the denoised image feature vector.

[0037] Specifically, refer to Figure 2 In the specific embodiment of the present invention, the image denoising network F D The two 3*3 convolutional layers at the feature input end and the feature concatenation module connecting the two 3*3 convolutional layers are used to enhance the feature fusion of the description matrix and the target image. The fused image feature vector is extracted and denoised through the sequentially connected feature extraction module, soft threshold denoising module and feature extraction module. Finally, a 3*3 convolutional layer is used to reduce the number of channels to the same denoised image feature vector as the target image. The structure of the feature extraction module of the image denoising network is the same as that of the feature extraction module in the image enhancement network.

[0038] Among them, the soft threshold denoising module is used to perform the following operations: (2) Where v is the characteristic value input to the soft threshold denoising module, that is, the image denoising network F D The first feature extraction module inputs the eigenvalues ​​in each feature matrix of the soft threshold denoising module. δ is the preset soft threshold parameter. When v is greater than 0, Sign(v) = 1, and when v is less than 0, Sign(v) = -1.

[0039] Among them, the preset soft threshold parameter δ can be set as needed. It can be seen from the calculation model of the soft threshold denoising module that when the absolute value of the eigenvalue (also called signal value) in the feature matrix is ​​greater than the soft threshold parameter, the eigenvalue is retained. If it is less than or equal to the soft threshold parameter, it proves that the eigenvalue is noise and the eigenvalue is set to zero. Therefore, by adjusting the threshold size, the noise in the signal can be effectively suppressed while retaining the main characteristics of the signal.

[0040] Furthermore, in step S4 of the embodiment of the present invention, performing image conversion on the denoised image feature vector based on the enhanced description matrix includes: using a preset image conversion model to realize image conversion on the denoised image feature vector based on the enhanced description matrix, and the image conversion model is expressed as: (3) Among them, p refers to the pixel point in the image, Φ L (p) refers to the eigenvalue of the enhanced description matrix at pixel p, L D (p) refers to the eigenvalue of the denoised image feature vector at pixel p, L E (p) refers to the calculated eigenvalue of the enhanced image feature vector at pixel point p.

[0041] Specifically, at the same pixel point p, the eigenvalue of the enhanced description matrix at the pixel point p is used to perform a power exponential operation on the eigenvalue of the denoised image eigenvector at the pixel point p to obtain the eigenvalue of the enhanced image eigenvector at the pixel point p.

[0042] For example, in low-brightness image areas, if the value range corresponding to the enhanced description matrix is ​​greater than 0 and less than 1, the image eigenvalue will increase, for example L D (p) The characteristic value of a pixel is 0.3. Φ L (p) The corresponding value is 0.75, 0.3 0.75 ≈ 0.4053 ; In the overly bright image area, if the value corresponding to the enhancement description matrix is ​​greater than 1, the image eigenvalue will become smaller, for example L D (p) A pixel value of is 0.9, Φ L (p) The corresponding value is 1.25, 0.9 1.25 ≈ 0.8766; This value is not within the scope of the present invention. The default image is a low-brightness image, and there is no overbrightness. In the image area with moderate brightness, the value corresponding to the enhancement description matrix is ​​equal to 1, and the image eigenvalue remains unchanged, for example L D (p) The characteristic value of a pixel is 0.5. Φ L (p) The corresponding value is 1, 0.5 1 = 0.5。

[0043] It should be noted here that the number of channels of the enhanced image feature vector is the same as the number of channels of the denoised image feature vector. For example, when the denoised image feature vector has three eigenvalues ​​at pixel point p, it is necessary to use the enhanced description matrix to perform power exponential operations on the eigenvalues ​​at pixel point p respectively, and finally obtain the three eigenvalues ​​of the enhanced image feature vector at pixel point p.

[0044] In an embodiment of the present invention, before using an image enhancement network to perform image enhancement on a target image, it is necessary to first use a sample data set containing low-light images and high-light images to iteratively optimize and train the image enhancement network, use the total loss function of the image enhancement network to calculate the total loss after each iteration, and continuously adjust the model parameters in the image enhancement network according to the total loss after each iteration until the total loss of the image enhancement network converges, so that the image enhancement network has a good image enhancement effect.

[0045] Furthermore, the embodiment of the present invention also requires training the model before performing image enhancement on the target image, and the method further includes: S01, constructing a training set, wherein the training set includes a group of multiple low-light images under low-light conditions and a group of multiple high-light images under good lighting conditions; In the embodiment of the present invention, constructing the training set specifically involves collecting two groups of digestive endoscopy images of the same number, one group including images L under low light conditions and one group including images H under good light conditions, and the contents of images L and H may be different. It should be noted that both images L and H in the embodiment of the present invention have been normalized, so that each pixel in the image has a normalized pixel value greater than 0 and less than 1.

[0046] Specifically, for digestive endoscopy images, the shooting parts of the digestive endoscopy images were counted. In order to ensure the balance of the number of training sets, the number of images in different parts of the digestive tract was kept consistent as much as possible, and duplicate images were deleted or images of parts with a smaller number were supplemented. After collecting the images, images L and H were placed in two folders respectively to distinguish them.

[0047] S02, fixing the model parameters in the image denoising network, performing multiple iterations of model training on the image enhancement network based on the training set, and calculating the total loss of image enhancement after each iteration of training according to the total loss function of the image enhancement network during the iterative training process, adjusting the model parameters in the image enhancement network according to the total loss of image enhancement, and stopping the iterative training after the total loss of image enhancement reaches an expected loss threshold; S03. Fix the model parameters in the image enhancement network, perform multiple iterative model training on the image denoising network based on the training set, and calculate the total image denoising loss after each iterative training according to the total loss function of the image denoising network during the iterative training process, adjust the model parameters in the image denoising network according to the total image denoising loss, and stop the iterative training until the total image denoising loss reaches an expected loss threshold.

[0048] The present invention trains the image enhancement network and the image denoising network separately, does not require manual annotation of data for training, and uses the characteristics of the data itself as a supervisory signal, so that the model can self-learn and extract useful features.

[0049] Furthermore, the total loss function of the image enhancement network provided by the embodiment of the present invention includes a self-supervised loss function of a preset first weight coefficient, and the self-supervised loss function is expressed as: (4) Among them, 0<first weight coefficient≤1, Loss s is the self-supervised loss, L refers to low-light images, and F M refers to the image enhancement network, F M (L) is the output result of the low-light image after being processed by the image enhancement network, ω is a custom parameter for self-supervised learning, 0<ω<1, Lω is the calculation result after optimizing the normalized pixel value of each pixel in L using the parameter ω, and F M (Lω) is the result of Lω being processed by the image enhancement network. The calculation formula of Lω is as follows: (5) Where L(p) is the normalized pixel value of a single pixel in the low-light image; For example, if the parameter ω=0.75 and the P value of a pixel in image L is 0.8, then after calculation by this formula, the value of the pixel becomes 0.8 0.75 ≈0.8458. Through the processing of parameter ω, the normalized pixel value of the image will become larger, which means that its brightness will increase. The purpose of Lω is to amplify the pixel value and increase the overall brightness of the image.

[0050] Since the self-supervised loss Los(L,ω) is calculated based on the image L and the parameter ω rather than two images with one-to-one pixel correspondence, the training set can use unpaired low-light digestive endoscopy images and high-light digestive endoscopy images instead of having to use pairs of digestive endoscopy images with exactly the same content.

[0051] Furthermore, in order to ensure the consistency of low-light image enhancement, a low-light image enhancement loss is designed. This loss ensures that the image L has been enhanced using F M The result after network processing, if F is performed again M After network processing, the output enhancement Map is a matrix whose elements are all 1, so that the image is no longer enhanced.

[0052] Based on the above design ideas, the total loss function of the image enhancement network can also include a low-light image enhancement loss function with a preset second weight coefficient. The low-light image enhancement loss function is expressed as: (6) Among them, 0<second weight coefficient≤1, Loss C is the low-light image enhancement loss, L M = F M (L), F M (L M ) is the result of the second processing of the low-light image by the image enhancement network after the image enhancement network has processed the output result. E is a matrix with the same scale as the enhancement description matrix and all elements are 1; Furthermore, due to F M (L M ) The generated enhanced description matrix has all elements 1, so that the low-light image L uses F M (L M ), any pixel value of the image remains unchanged. For example, the normalized pixel value of the image is 0.6. M (L M ) after processing 0.6 1 = 0.6. This ensures that the image L has been used F M The result after network processing, if F is performed again M After network processing, the brightness remains unchanged, which effectively avoids secondary enhancement of low-light digestive endoscopy images.

[0053] Furthermore, in order to ensure the consistency of the high-light image enhancement effect, the present invention designs a high-light image enhancement loss. This loss ensures that the image H is M The enhanced Map output by the network is a matrix with all elements set to 1, so that the highlight image is not enhanced.

[0054] Based on the above design ideas, the total loss function of the image enhancement network can also include a high-light image enhancement loss function with a preset third weight coefficient. The high-light image enhancement loss function is expressed as: (7) Among them, 0<third weight coefficient≤1, Loss H is the highlight image enhancement loss, H is the highlight image, and F M (H) is the output result of the highlight image after being processed by the image enhancement network.

[0055] Because F M The elements of (H) are all 1, so that in F M (H), any pixel value of the high-light image remains unchanged. For example, the image pixel value is 0.9, and after enhancement processing, 0.9 1 = 0.9, which ensures that image H uses F M The brightness of the result after network processing remains unchanged, which effectively avoids enhancing the high-light digestive endoscopy image.

[0056] In a specific embodiment of the present invention, the total loss function of the image enhancement network is expressed as: (8) Among them, Loss s Refers to self-supervision loss, Loss C Refers to low-light image enhancement loss, Loss H Refers to the high-light image enhancement loss, α and β represent weight parameters, which can be adjusted according to the effect. In a specific embodiment of the present invention, α and β are both 0.02; The present invention utilizes the characteristics of the data itself as a supervisory signal, so that the model can self-learn and extract useful features. Therefore, there is no need to use exactly the same low-light images and high-light images for model training, which simplifies the model training process.

[0057] Furthermore, before using convolutional layers to calculate the target image and the enhanced description matrix to form feature vectors, the method also includes: performing model training on the image denoising network based on a preset total loss function of the image denoising network.

[0058] In the embodiment of the present invention, the total loss function of the image denoising network is obtained by weighted addition of the low-light image denoising loss and the high-light image denoising loss; the total loss function of the image denoising network is expressed as: (9) Among them, Loss D (L) refers to the low-light image denoising loss, Loss D(H) refers to the denoising loss of high-light images, L refers to low-light images, H refers to high-light images, and k and z represent weight parameters.

[0059] Furthermore, in order to ensure that the low-light image does not lose the details of the original image content after denoising, a low-light image denoising loss is designed. The low-light image denoising loss is expressed as: (10) Among them, Loss D (L) refers to the low-light image denoising loss, F D (L) is the output result of the low-light image after being processed by the image denoising network.

[0060] Furthermore, in order to ensure that the highlight image does not lose the details of the original image content after denoising, a highlight image denoising loss is designed. The highlight image denoising loss is expressed as: (11) Among them, Loss D (H) refers to the denoising loss of high-light images, F D (H) is the output result of the high-light image after being processed by the image denoising network.

[0061] Furthermore, in order to verify the feasibility of this method, in a specific embodiment of the present invention, an experiment was conducted using 4000 digestive endoscopy images, in which the training set, validation set, and test set were divided into 70%, 20%, and 10% ratios. The training set used a group of 1400 low-light digestive endoscopy images and a group of 1400 high-light digestive endoscopy images, and the two groups of images were non-matching image pairs. The model training is divided into two stages. The first stage only trains F M After completing the training, fix the network weight parameters of this module and then train F D After completing the two-stage training of the model, the entire network model was used for experiments. Some of the experimental results are shown in Figure 3 As shown, Figure 3 (a) is a low-light endoscopic image of the stomach. Figure 3 (b) is the enhanced gastric endoscopic image. Figure 3 (c) is a low-light endoscopic image of the colon. Figure 3 (d) in the figure is the enhanced colon endoscope image. This method has a significant enhancement effect on the low-light image of the digestive endoscope, retaining the original image details and effectively improving the image brightness.

[0062] The present invention establishes a low-light image enhancement method for digestive tract endoscope based on self-supervised learning. Figure 2The network model shown completes image brightness enhancement and denoising. The model has the following advantages: 1) The training set uses unpaired low-light digestive endoscopy images and high-light digestive endoscopy images, which are very easy to collect; 2) It effectively avoids enhancing high-light digestive endoscopy images and secondary enhancing low-light digestive endoscopy images; 3) It effectively handles the noise in the image and improves the image quality. The present invention helps to improve the quality of digestive endoscopy images, can effectively improve the effect of digestive endoscopy examination, and has high clinical and scientific research application value.

[0063] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0064] Embodiment 2 Figure 3 The schematic diagram of the structure of a low-light digestive endoscope image enhancement device provided by an embodiment of the present invention is shown schematically. Figure 4 A low-light digestive endoscope image enhancement device according to an embodiment of the present invention specifically includes an image processing module 401, an image enhancement module 402, an image denoising module 403 and an image conversion module 404, wherein: An image processing module 401 is used to normalize the pixel value of each pixel in the initial target image to be enhanced to obtain a target image; An image enhancement module 402 is used to extract features of a target image based on a preset image enhancement network to identify brightness features of each pixel in the target image, and obtain an enhancement description matrix, wherein the enhancement description matrix is ​​a two-dimensional matrix with the same scale as the target image; An image denoising module 403 is used to obtain feature vectors using convolution layers for the target image and the enhanced description matrix, perform feature fusion on the feature vectors to obtain a fused image feature vector, perform feature extraction on the fused image feature vector, and perform denoising on the extracted feature values ​​to obtain a denoised image feature vector; An image conversion module 404 is used to perform image conversion on the denoised image feature vector based on the enhancement description matrix, so as to enhance the feature value of the denoised image feature vector at each pixel point to different degrees according to the brightness feature of the corresponding pixel point, thereby obtaining an enhanced image feature vector; The image processing module 401 is further configured to perform a denormalization operation on the feature value of each pixel point of the enhanced image feature vector to obtain a target enhanced image of the initial target image.

[0065] Furthermore, the image conversion module 404 is specifically used to implement image conversion of the denoised image feature vector based on the enhanced description matrix by using a preset image conversion model. The image conversion model is expressed as: Furthermore, the image enhancement module 402 is specifically used to perform corresponding image enhancement operations on the input image enhancement network based on a preset image enhancement network, wherein the operation process of the image enhancement network is the same as the operation process in the method embodiment, and will not be repeated here.

[0066] Furthermore, the embodiment of the present invention further includes a network training module, which is used to perform model training on the image enhancement network based on a preset total loss function of the image enhancement network before the image enhancement module 402 extracts features from the target image to identify the brightness features of each pixel in the target image. The total loss function of the image enhancement network is the same as that in the method embodiment, and will not be described in detail here.

[0067] Furthermore, the image denoising module 403 is specifically used to perform corresponding image denoising operations on the target image and the enhanced description matrix input into the image denoising network based on a preset image denoising network, wherein the operation process of the image denoising network is the same as the operation process in the method embodiment, and will not be repeated here.

[0068] Furthermore, the network training module of the embodiment of the present invention is also used to perform model training on the image denoising network based on a preset total loss function of the image denoising network before the image denoising module 403 performs feature fusion on the target image and the enhanced description matrix to obtain a fused image feature vector. The total loss function of the image denoising network is the same as that in the method embodiment and will not be described in detail here; The embodiment of the present invention provides a method and device for low-light digestive endoscope image enhancement. The method normalizes the pixel values ​​of each pixel in an initial target image to be enhanced to obtain a target image, and extracts features from the target image to identify the brightness features of each pixel in the target image to obtain an enhanced description matrix; at the same time, convolutional layers are used to obtain feature vectors for the enhanced description matrix and the target image respectively, and image fusion is performed on the feature vectors to obtain a fused image feature vector, and feature extraction is performed on the fused image feature vector, and a denoising operation is performed on the extracted feature values ​​to obtain a denoised image feature vector; finally, based on the enhanced description matrix, image conversion is performed on the denoised image feature vector to obtain an enhanced image feature vector. Since the enhanced description matrix can reflect the brightness features of each pixel, when the denoised image is converted, the feature values ​​of the denoised image feature vector of the pixel with lower brightness can be highly enhanced, thereby achieving the purpose of improving image quality and image enhancement for low-light images at the same time.

[0069] Embodiment 3 An embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned embodiments of the method for enhancing the image of a low-light digestive endoscope are implemented, for example: Figure 1 Alternatively, when the processor executes the computer program, the functions of each module / unit in the above-mentioned low-light image enhancement device embodiments are realized, for example Figure 4 The image processing module 401, the image enhancement module 402, the image denoising module 403 and the image conversion module 404 are shown.

[0070] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the claimed embodiments may be used in any combination.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A low-light digestive endoscope image enhancement method, characterized in that: The method comprises: Normalizing the pixel value of each pixel in the initial target image to be enhanced to obtain the target image; Based on a preset image enhancement network, feature extraction is performed on the target image to identify the brightness features of each pixel in the target image, and an enhancement description matrix is ​​obtained, wherein the enhancement description matrix is ​​a two-dimensional matrix with the same scale as the target image; Using convolutional layers to calculate feature vectors for the target image and the enhanced description matrix respectively, performing feature fusion on the feature vectors to obtain a fused image feature vector, performing feature extraction on the fused image feature vector, and performing a denoising operation on the extracted feature values ​​to obtain a denoised image feature vector; Performing image transformation on the denoised image feature vector based on the enhanced description matrix, so as to enhance the feature value of the denoised image feature vector at each pixel point to different degrees according to the brightness characteristics of the corresponding pixel point, thereby obtaining an enhanced image feature vector; A denormalization operation is performed on the eigenvalues ​​of each pixel point of the enhanced image eigenvector to obtain a target enhanced image of the initial target image.

2. The method according to claim 1, characterized in that The image conversion of the denoised image feature vector based on the enhanced description matrix includes: The preset image conversion model is used to realize the image conversion of the denoised image feature vector based on the enhanced description matrix. The image conversion model is expressed as: Among them, p refers to the pixel point in the image, Φ L (p) refers to the eigenvalue of the enhanced description matrix at pixel p, L D (p) refers to the eigenvalue of the denoised image feature vector at pixel p, L E (p) refers to the calculated eigenvalue of the enhanced image feature vector at pixel point p.

3. The method according to claim 1, characterized in that The image enhancement network is used to complete the following operations: Iteratively extracting features of the target image through a plurality of feature extraction modules connected in sequence, and obtaining a feature matrix of a plurality of channels for representing brightness features; A convolution operation is performed on the feature matrices of multiple channels to obtain a two-dimensional matrix of one channel, and the obtained two-dimensional matrix is ​​used as an enhanced description matrix.

4. The method according to claim 1, characterized in that Before extracting features of the target image based on a preset image enhancement network to identify brightness features of each pixel in the target image, the method further includes: The image enhancement network is model trained based on a preset total loss function of the image enhancement network, wherein the total loss function of the image enhancement network includes a self-supervised loss function of a preset first weight coefficient, and the self-supervised loss function is expressed as: Among them, 0<first weight coefficient≤1, Loss s is the self-supervised loss, L refers to low-light images, and F M refers to the image enhancement network, F M (L) is the output result of the low-light image after being processed by the image enhancement network, ω is a custom parameter for self-supervised learning, 0<ω<1, Lω is the calculation result after optimizing the normalized pixel value of each pixel in L using the parameter ω, and F M (Lω) is the result of Lω being processed by the image enhancement network. The calculation formula of Lω is as follows: Among them, L(p) is the normalized pixel value of a single pixel in the low-light image.

5. The method according to claim 4, characterized in that The total loss function of the image enhancement network also includes a low-light image enhancement loss function of a preset second weight coefficient, and the low-light image enhancement loss function is expressed as: Among them, 0<second weight coefficient≤1, Loss C is the low-light image enhancement loss, L M = F M (L), F M (L M ) is the result of the second processing of the low-light image by the image enhancement network after the image enhancement network has processed the output result. E is a matrix with the same scale as the enhancement description matrix and all elements are 1.

6. The method according to claim 4 or 5, characterized in that: The total loss function of the image enhancement network also includes a high-light image enhancement loss function of a preset third weight coefficient, and the high-light image enhancement loss function is expressed as: Among them, 0<third weight coefficient≤1, Loss H is the highlight image enhancement loss, H is the highlight image, and F M (H) is the output result of the highlight image after being processed by the image enhancement network.

7. The method according to claim 1, characterized in that The method of using convolution layers to obtain feature vectors for the target image and the enhanced description matrix respectively, performing feature fusion on the feature vectors to obtain a fused image feature vector, performing feature extraction on the fused image feature vector, and performing a denoising operation on the extracted feature values ​​to obtain a denoised image feature vector comprises: Based on the preset image denoising network, the following operations are performed on the target image and the enhanced description matrix input into the image denoising network: Performing a convolution operation on the target image to obtain a first convolution input image feature vector; Performing a convolution operation on the enhanced description matrix to obtain a second convolution enhanced description feature vector; Performing a concatenation operation on the first convolution input image feature vector and the second convolution enhancement description feature vector to obtain a fused image feature vector of the target image and the enhancement description matrix; At least one feature extraction module and at least one soft threshold denoising module respectively perform feature extraction operations and soft threshold denoising operations on the fused image feature vector to obtain an initial denoised image feature vector; wherein the soft threshold denoising module is used to perform the following operations: Wherein, v is the characteristic value input to the soft threshold denoising module, δ is the preset soft threshold parameter, when v is greater than 0, Sign(v) = 1, when v is less than 0, Sign(v) = -1; A convolution operation is performed on the initial denoised image feature vector to obtain at least one feature matrix having the same number of channels and scale as the target image, and the obtained at least one feature matrix is ​​used as the denoised image feature vector.

8. The method according to claim 7, characterized in that Before respectively using convolutional layers to calculate the target image and the enhanced description matrix to form feature vectors, the method further includes: Performing model training on the image denoising network based on a preset total loss function of the image denoising network; The total loss function of the image denoising network is obtained by weighted addition of the low-light image denoising loss and the high-light image denoising loss; wherein, The low-light image denoising loss function is expressed as: Among them, Loss D (L) refers to the low-light image denoising loss, F D (L) is the output result after the low-light image is processed by the image denoising network, L refers to the low-light image; The high light image denoising loss is expressed as: Among them, Loss D (H) refers to the denoising loss of high-light images, F D (H) is the output result after the highlight image is processed by the image denoising network, and H is the highlight image.

9. The method according to claim 3 or 7, characterized in that: The feature extraction module is used to complete the following operations: Performing iterative convolution feature extraction on the image features of the input feature extraction module through multiple feature extraction convolution layers; In the process of convolutional feature extraction, the weight of channel features is adaptively adjusted through the preset channel attention module to increase the weight of the channel used to characterize brightness; The image features obtained after performing convolution feature extraction and the image features input to the feature extraction module are element-wise added to obtain the output features of the feature extraction module.

10. A low-light digestive endoscope image enhancement device, characterized in that: The device comprises: An image processing module is used to normalize the pixel value of each pixel in the initial target image to be enhanced to obtain a target image; An image enhancement module is used to extract features of a target image based on a preset image enhancement network to identify brightness features of each pixel in the target image, and obtain an enhancement description matrix, wherein the enhancement description matrix is ​​a two-dimensional matrix with the same scale as the target image; An image denoising module, used to obtain feature vectors using convolution layers for the target image and the enhanced description matrix respectively, perform feature fusion on the feature vectors to obtain a fused image feature vector, perform feature extraction on the fused image feature vector, and perform denoising operation on the extracted feature values ​​to obtain a denoised image feature vector; An image conversion module is used to perform image conversion on the denoised image feature vector based on the enhancement description matrix, so as to enhance the feature value of the denoised image feature vector at each pixel point to different degrees according to the brightness characteristics of the corresponding pixel point, thereby obtaining an enhanced image feature vector; The image processing module is further used to perform a denormalization operation on the eigenvalues ​​of each pixel point of the enhanced image eigenvector to obtain a target enhanced image of the initial target image.

11. A computer device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the low-light digestive endoscopy image enhancement method as described in any one of claims 1 to 9 is implemented.

Images