Capsule endoscope low-light image detail enhancement and color correction method

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a method for detail enhancement and color correction of a low-light image of a capsule endoscope. The method comprises the following steps: constructing a lightweight low-light enhanced convolutional neural network model; a low-light image collected by the capsule endoscope is processed by the network model, and a normal image is recovered; the network model comprises three feed-forward neural network blocks (FFN) connected in series; each FFN comprises two image guide enhancement modules (IGEBs) and a 3 * 3 convolution; each IGEB comprises three image guide enhancement functions, four transformation convolution residual blocks (TCRB), two matrix color correction modules (MCC) and a 3 * 3 convolution; an MCC and a 3 * 3 convolution are arranged at the last of the network model; the neural network model constructed by the invention can effectively recover the detail information of the low-light image, correct the color distortion problem generally existing in the low-light image, and well reserve and enhance the structure and color information of the original image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for detail enhancement and color correction of a capsule endoscope low-light image. Background Art

[0002] Wireless capsule endoscopy (WCE) is a revolutionary technology in the field of gastrointestinal diagnosis, with significant advantages over traditional endoscopy. By swallowing a small capsule equipped with a camera, patients can non-invasively image the entire gastrointestinal tract, especially the small intestine that was previously difficult to visualize. However, the complexity of the gastrointestinal structure, coupled with insufficient ambient light and camera performance limitations, may result in underexposed images. However, early lesions may be difficult to detect under low-light conditions, so there is an increasing demand for low-light medical image enhancement technology. In recent years, deep learning technology has made significant progress in the field of low-light image enhancement [1][2]. However, some enhancement methods may introduce color distortion, especially when processing low-light images, maintaining color authenticity is a challenge [2]. The image may also lose some important detail information, such as texture and edges, during the enhancement process, which affects subsequent image analysis and diagnosis [1]. Summary of the invention

[0003] The object of the present invention is to provide a method for detail enhancement and color correction of capsule endoscope low-light images based on a lightweight neural network, so as to well restore the color of the image brightness and retain the detail information of the original image.

[0004] The method for detail enhancement and color correction of capsule endoscope low-light images based on lightweight neural network provided by the present invention mainly includes constructing a lightweight low-light enhancement convolutional network model, taking the low-light image collected by the capsule endoscope as the input of the network model, and restoring the normal image. The model input image first enters a 3x3 convolution, then enters three feedforward neural network blocks (FFN) in sequence, and finally outputs the restored image after passing through a matrix color correction module (MCC) [5] and a 3x3 convolution. In each FFN, two image guided enhancement modules (IGEB) and a 3x3 convolution are included in series. In IGEB, the input low-light image is first concatenated after passing through three independent enhancement functions, and then the feature map obtained by a 1x1 convolution is concatenated with the output of the previous module as the IGEB input. IGEB has two transform convolution residual blocks (TCRBs) and an MCC connected in sequence, and then two TCRBs and an MCC are connected in sequence, and the last 3x3 convolution is processed as the IGEB output. In each TCRB, the input first passes through a 1x1 convolution, and then is divided into two branches: the spatial convolution branch and the transform kernel convolution branch. In the spatial convolution branch, two groups of 1x1 convolutions, 3x3 depthwise separable convolutions, and PReLu activation functions are connected in sequence; in the transform kernel convolution branch, 1x1 convolutions, transform kernel convolutions, 1x1 convolutions, 3x3 depthwise separable convolutions, and PReLu activation functions are connected in sequence. Finally, the results of the two branches are concatenated as the output of the TCRB. See Figure 1 shown.

[0005] (1) Image-guided enhancement module (IGEB):

[0006] After applying the image enhancement function to the original image, it is sent to each layer of the network as a guide to retain the rich detail information in the original image. Using only one enhancement function at each stage may limit the network to learn a set of globally optimal function parameters. Since it is necessary to consider the local details of the input image, three independent enhancement functions are used in each IGEB, and these enhanced images are fused and merged with the feature maps of different stages of the network to guide the network. After the input image passes through three independent enhancement functions, the output images are spliced ​​together. Then a 1×1 convolution is used to obtain the feature map M 1 . M 2 is the output feature map of the previous block, and M 1 With M 2 After splicing, it is used as the final input of the image-guided enhancement module, see Figure 1 As shown. The whole process can be described by the following equation:

[0007] M 1 =O c (Concat(f(I;θ 1 ), f(I;θ2 ), f(I;θ 3 ))), (1)

[0008] IGEB input =Concat(M 2 ,M 1 ), (2)

[0009] Where f refers to the image-guided enhancement function, I is the input image, and θ i (i=1,2,3) are the enhancement function parameters; O C Refers to the convolution operation, and Concat refers to the concatenation of feature maps.

[0010] When designing the guided enhancement function in formula (1), the different characteristics of the brightness and chromaticity components are taken into consideration. Different enhancement methods are designed for the brightness and chromaticity components: for the brightness component, a logarithmic curve is used to solve the darkness problem. The input image I is decomposed into three components: Y, U, and V. The parameter θ in formula (1) i It can be further decomposed into [λ i1 ,λ i2 ,λ i3 ,ε] four learnable parameters. For the brightness component, by combining the learnable parameter λ i1 ,ε,The logarithmic curve can effectively enhance the contrast of low-brightness areas while retaining the details of high-brightness areas.

[0011] For the chrominance components U and V, the parameters λ are introduced respectively. i2 , i3 To make up for the lack of chromaticity components of low-light images. The whole method can be written as follows:

[0012]

[0013] Among them, I e The result of the image after the enhancement function, Y e , U e 、V e and Y, U, and V refer to the enhanced image I e and the YUV components of the original image I, λ ij (i=1,2,3) and ε are all learnable parameters in the function during training, max(Y) refers to the maximum value of the Y component, and log refers to the logarithmic function.

[0014] (2) Transformed Convolutional Residual Block (TCRB):

[0015] Transformed Convolution Residual Block (TCRB), characterized in that a transform kernel (TFK) is designed in TCRB to extract high-frequency details of feature maps. After the feature map is input into the Transformed Convolution Residual Block, it first undergoes a 1x1 convolution, and then is mainly divided into two parts: the top spatial convolution branch and the bottom transform kernel convolution branch.

[0016] First, the input feature map of TCRB is fed into a 1x1 convolutional layer:

[0017] X c =O C (X) , (4)

[0018] Among them, X is the input of TCRB, X c is the output of the convolution; next, X c Send to two different branches, X s It is the spatial convolution branch O spatial The output result, X tf is the transformation kernel convolution branch O transform The output is:

[0019] X s =O spatial (X c ), (5)

[0020] X tf =O transform (X c ), (6)

[0021] Operation O in formula (5) spatial It can be further decomposed into several operations and written as:

[0022] X s =O CDP (O CDP (X c ))+X c (7)

[0023] Among them, O CDP Refers to a series of operations including a 1x1 convolution, a 3x3 depth-wise convolution, and a PReLu activation function.

[0024] Operation O in formula (6) transform It can be further decomposed into several operations and written as:

[0025] X tf =O CDP (O CT (X c)) (8)

[0026] Among them, O CT Refers to a series of operations including a 1x1 convolution and a transform convolution. s and X tf The final output of the transformed convolution residual block is obtained by splicing:

[0027] TCRB output =Concat(X s ,X tf ) (9)

[0028] For the transform kernel convolution branch, the Laplacian kernel is selected as the basic component of the transform kernel, which helps the transform convolution residual module to learn the high-frequency characteristics of the feature map more deeply, so that the network can capture finer details of the feature map. The original form of the Laplacian kernel is:

[0029]

[0030] This is a two-dimensional filter kernel, so the filter kernel is replicated in the channel dimension, and finally a convolution kernel of shape C×W×H is obtained as the final transformation kernel, where C is the number of feature map channels, W is the width of the feature map, and H is the height of the feature map. For details, see Figure 1 shown.

[0031] (3) Matrix color correction module:

[0032] The matrix color correction module (MCC) [5], after layer normalization, 1x1 convolution and 3x3 depth-wise convolution, obtains a feature map with C channels, W width and H height. The feature map is then transformed into three matrices Q, K and V with a shape of HW×C, and then a matrix transformation color correction operation Q×K is performed. T ×V, a new matrix of shape HW×C is obtained, which is reshaped into a feature map with C channels, W width, and H height. After 1x1 convolution, it is added to the residual of the MCC input feature map, and finally output after 1x1 convolution and 3x3 depth-separable convolution.

[0033] The technical features and performance advantages of the present invention are mainly:

[0034] A lightweight convolutional network for end-to-end low-light enhancement of gastrointestinal endoscope images, which includes a novel transformation kernel convolution and image-guided enhancement module, can effectively restore the detail information of low-light images and correct the color distortion problem that usually exists in low-light images. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Illustration of the structure of the lightweight low-light enhanced convolutional neural network model constructed for the present invention.

[0036] Figure 2 Figure 1 is a set of Red Lesion dataset processing examples, which include, from left to right, gastrointestinal endoscope low-light images, the results of references [2][1], the results of the present invention, and the corresponding benchmark images.

[0037] Figure 3 Figure 1 is a set of Kvasir-Capsule dataset processing examples, which include, from left to right, gastrointestinal endoscopy low-light images, the results of references [2][1], the results of the present invention, and the corresponding benchmark images. DETAILED DESCRIPTION

[0038] The method for capsule endoscope low-light image detail enhancement and color correction based on lightweight neural network provided by the present invention comprises the following specific steps:

[0039] (1) Dataset determination

[0040] We use two publicly available gastrointestinal endoscopy datasets from LLCaps[1]: the Kvasir-Capsule dataset (KCL)[3] and the Red Lesion dataset (RLE)[4]. The KCL dataset contains a paired training set of 2,000 images and a paired test set of 400 images. The RLE dataset has a paired training set of 946 images and a paired test set of 334 images.

[0041] (2) Determine the training method

[0042] The training sets of the two datasets are combined to train the model. The entire model is trained end-to-end. The model is trained for 150 rounds using the Adam optimizer, with a batch size of 6 and a learning rate of 5×10 -3 .

[0043] (3) Determination of loss function

[0044] Charbonnier loss is used as the loss function:

[0045]

[0046] Among them, I output Refers to the network model output, I gt Refers to the baseline image corresponding to the low-light image, and the ε value is set to 1×10 -3 .

[0047] (4) Training process

[0048] (4.1) The low-light RGB image is normalized and used as the input of the model.

[0049] (4.2) The input first undergoes a 3x3 convolution to increase the number of channels to 64.

[0050] (4.3) Next, three feedforward neural network blocks (FFN) are passed in sequence, where each feedforward neural network block includes two image guided enhancement modules (IGEB).

[0051] (4.4) Each image-guided enhancement module contains three image-guided enhancement functions, four transform convolution residual blocks (TCRBs), two matrix color correction modules (MCCs), and a 3x3 convolution. In the image-guided enhancement module, the input low-light image is enhanced by three independent enhancement functions and then concatenated to obtain a 9-layer feature map, and a 64-layer feature map is obtained through 1x1 convolution ( Figure 1 M 1 ), and then output the feature map of the previous module ( Figure 1 M 2 ) are concatenated to obtain a 128-layer feature map as the input of the transform convolutional residual block (TCRB).

[0052] (4.5) In the transform convolution residual block, the input feature map first undergoes a 1x1 convolution to change the number of feature map channels from 128 layers to 64 layers, and then passes through two different branches in parallel: the spatial convolution branch and the transform kernel convolution branch. The results of the two branches are concatenated as the output of the transform convolution residual block.

[0053] (4.6) Finally, the feature map passes through a matrix color correction module (MCC) [5] and a 3x3 convolution to output a high-quality image after low-light enhancement.

[0054] (5) Reasoning process

[0055] For a trained model, you only need to input the normalized low-light RGB image during inference, and the model will produce a restored enhanced image.

[0056] (6) Test results

[0057] The present invention finally obtains the experimental results as follows:

[0058]

[0059] On the Kvasir-Capsule dataset [3], the peak signal-to-noise ratio (PSNR) of the present invention reaches 39.17; on the RedLesion dataset [4], the PSNR of the present invention reaches 33.32. Compared with methods [1] and [2], the model constructed by the present invention can better restore the details and color information of the image and obtain better image quality.

[0060] References

[0061] [1].Bai L,Chen T,Wu Y,et al.Llcaps:Learning to illuminate low-lightcapsule endoscopy with curved wavelet attention and reverse diffusion[C] / / International Conference on Medical Image Computing and Computer-AssistedIntervention.Cham:Springer Nature Switzerland,2023:34-44.

[0062] [2]. Zamir SW, Arora A, Khan S, et al. Learning enriched features for fast image restoration and enhancement [J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 45(2): 1934-1948.

[0063] [3].Pia H.Smedsrud,Vajira Thambawita,Steven A.Hicks,Henrik Gjestang,Oda Olsen Nedrejord,Espen Hanna Borgli,Debesh Jha,Tor Jan Derek Berstad,Sigrun L.Eskeland,Mathias Lux,Havard Espeland,°Andreas Petlund,Duc Tien DangNguyen,Enrique Garcia-Ceja,Dag Johansen,Peter T.Schmidt,Ervin Toth,HugoL,Hammer,Michael Lang. A.Riegler,and Pal Halvorsen,“Kvasir-capsule,°a video capsule endoscopy dataset,”Scientific Data,vol.8,no.1,pp.142,May 2021.

[0064] [4].Paulo Coelho,Ana Pereira,Argentina Leite,Marta Salgado,andAntonio′Cunha,“A deep learning approach for red lesions detection in videocapsule endoscopies,”in Image Analysis and Recognition,Aurelio′Campilho,Fakhri Karray,and Bart ter Haar Romeny,Eds.,Cham,2018,pp.553–561,SpringerInternational Publishing.

[0065] [5].Xin Jin,Ling-Hao Han,Zhen Li,Chun-Le Guo,Zhi Chai,and Chongyi Li,“Dnf:Decouple and feedback network for seeing in the dark,”in 2023IEEE / CVFConference on Computer Vision and Pattern Recognition(CVPR),2023,pp.

Claims

1. A method for capsule endoscope low-light image detail enhancement and color correction, characterized in that: The invention includes constructing a lightweight low-light enhanced convolutional network model, taking the low-light image collected by the capsule endoscope as the input of the network model, and restoring the normal image; the input image first enters a 3x3 convolution, and then enters three feedforward neural network blocks (FFN) in sequence, and finally outputs the restored image after passing through a matrix color correction module (MCC) and a 3x3 convolution; in each feedforward neural network block (FFN), two image guided enhancement modules (IGEB) and a 3x3 convolution are included in series; in the image guided enhancement module (IGEB), the input low-light image is first concatenated after passing through three independent enhancement functions, and then a 1x1 convolution is performed to obtain a feature map, which is concatenated with the output of the previous module as the input of the image guided enhancement module (IGEB); there are 2 in the image guided enhancement module (IGEB) A transform convolution residual block (TCRB) and a matrix color correction module (MCC) are connected in sequence, and then two transform convolution residual blocks (TCRB) and a matrix color correction module (MCC) are connected in sequence, and the last 3x3 convolution is processed as the output of the image guided enhancement module (IGEB); in each transform convolution residual block (TCRB), the input first passes through a 1x1 convolution, and then is divided into two branches: the spatial convolution branch and the transform kernel convolution branch; in the spatial convolution branch, two groups of 1x1 convolutions, 3x3 depth-separable convolutions, and PReLu activation functions are connected in sequence; in the transform kernel convolution branch, 1x1 convolutions, transform kernel convolutions, 1x1 convolutions, 3x3 depth-separable convolutions, and PReLu activation functions are connected in sequence; finally, the results of the two branches are spliced ​​as the output of TCRB.

2. The method for capsule endoscope low-light image detail enhancement and color correction according to claim 1, characterized in that: In the image-guided enhancement module (IGEB), after applying the image enhancement function to the original image, it is sent to each layer of the network as a guide to retain the rich detail information in the original image; three independent enhancement functions are used in each image-guided enhancement module (IGEB), and these enhanced images are fused and merged with the feature maps of different stages of the network to guide the network; after the input image passes through the three independent enhancement functions, the output images are spliced ​​together; then a 1×1 convolution is used to obtain the feature map M1; let M2 be the output feature map of the previous block, and splice M1 and M2 as the final input of the image-guided enhancement module. The whole process can be described by the following equation: M1=O C (Concat(f(I;θ1),f(I;θ2),f(I;θ3))), IGEB input =Concat(M2,M1), (2) Where f refers to the image-guided enhancement function, I is the input image, and θ i is the enhancement function parameter, i=1,2,3; O C Refers to the convolution operation, and Concat refers to the concatenation of feature maps; When designing the guided enhancement function in equation (1), different enhancement methods are designed for the luminance and chrominance components taking into account the different characteristics of the luminance and chrominance components: For the brightness component, a logarithmic curve is used to solve the darkness problem; the input image is decomposed into three components: Y, U, and V. The parameter θ in formula (1) i Further decomposed into [λ i1 ,λ i2 ,λ i3 ,ε] four learnable parameters; For the brightness component, by combining the learnable parameter λ i1 ,ε,The logarithmic curve can effectively enhance the contrast of low-brightness areas while retaining the details of high-brightness areas; For the chrominance components U and V, the parameters λ are introduced respectively. i2 , i3 To make up for the lack of chromaticity components of low-light images; the whole method is written in the following form: Among them, I e The result of the image after the enhancement function, Y e , U e 、V e and Y, U, and V refer to the enhanced image I e and the YUV components of the original image I, λ ij and ε are learnable parameters in the function during training, = 1, 2, 3, max(Y) refers to the maximum value of the Y component, and log refers to the logarithmic function.

3. The method for capsule endoscope low-light image detail enhancement and color correction according to claim 2, characterized in that: In the transform convolution residual block (TCRB), a transform kernel (TFK) is designed to extract high-frequency details of the feature map; after the feature map is input into the transform convolution residual block, it first undergoes a 1x1 convolution, and then is divided into two parts: a spatial convolution branch at the top and a transform kernel convolution branch at the bottom; First, the input feature map of the Transformed Convolutional Residual Block (TCRB) is fed into a 1x1 convolutional layer, expressed as: X c =O C (X), (4) Among them, X is the input of the Transformed Convolutional Residual Block (TCRB), X c is the output of the convolution; next, X c Send to two different branches, X s It is the spatial convolution branch O spatial The output result, X tf is the transformation kernel convolution branch O transform The output result is expressed as: X s =O spatial (X c ), (5) X tf =O transform (X c ), (6) Replace the operation O in formula (5) spatial Further decomposed into several operations, written as: X s =O CDP (O CDP (X c ))+X c (7) Among them, O CDP Refers to a series of operations: a 1x1 convolution, a 3x3 depth-wise separable convolution, and a PReLu activation function; Replace the operation O in formula (6) transform Further decomposed into several operations, written as: X tf =O CDP (The CT (X c )) (8) Among them, O CT Refers to a series of operations: a 1x1 convolution and a transform kernel convolution; X s and X tf The final output of the transformed convolution residual block is obtained by concatenation, which is expressed as: TCRB output =Concat(X s ,X tf ) (9)。 4. The method for capsule endoscope low-light image detail enhancement and color correction according to claim 3, characterized in that: For the transform kernel convolution branch, the Laplace kernel is selected as the basic component of the transform kernel; the original form of the Laplace kernel is: This is a two-dimensional filter kernel. By copying this filter kernel in the channel dimension, we finally get a convolution kernel of shape C×W×H as the final transformation kernel, where C is the number of feature map channels, W is the width of the feature map, and H is the height of the feature map.

5. The method for capsule endoscope low-light image detail enhancement and color correction according to claim 4, characterized in that: In the matrix color correction module (MCC), the input feature map is subjected to layer normalization, 1x1 convolution and 3x3 depth-separable convolution to obtain a feature map with C channels, W width and H height. The feature map is then transformed into three matrices Q, K and V with a shape of HW×C, and then a matrix transformation color correction operation Q×K is performed. T ×V, a new matrix of shape HW×C is obtained, which is reshaped into a feature map with C channels, W width, and H height. After 1x1 convolution, it is added to the residual of the MCC input feature map, and finally output after 1x1 convolution and 3x3 depth-separable convolution.

Images