Image quality enhancement method and system for endoscopic image
By combining complex and lightweight image quality enhancement models, using intermediate features, contrast and reconstruction losses to train lightweight models, and using edge-keeping filtering algorithms, the problem of slow endoscopic image processing speed is solved, achieving efficient image resolution Improve and significantly improve the efficiency of doctors' reading of films.
Patent Information
- Application Number
- PCT/CN2024/091037
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-04
- Filing Date
- 2024-04-30
- Publication Date
- 2024-11-07
AI Technical Summary
The existing endoscopic image quality enhancement model runs slowly, which cannot effectively improve the efficiency of doctors' reading, and cannot be optimized specifically for endoscopic images.
Using a combination of complex image quality enhancement models and light image quality enhancement models, train lightweight models through intermediate feature loss, contrast loss and reconstruction loss to improve the resolution processing capability of the model, and generate blurred images through edge-keeping filtering algorithm to optimize Model performance.
It significantly improves the resolution processing capability of endoscopic images, shortens the image processing time, and reduces it from a few hours to within one hour, greatly improving the efficiency of doctors' reading.
Smart Images

Figure CN2024091037_07112024_PF_FP_ABST
Abstract
Description
Endoscopic image quality enhancement method and system
[0001] This application claims priority to a Chinese patent application filed on May 4, 2023, with application number 202310484368.9, entitled “Method and device for enhancing the quality of endoscopic images,” the entire contents of which are incorporated herein by reference. Technical Field
[0002] The present invention relates to the field of endoscopic medical treatment, and in particular to a method and system for enhancing the image quality of an endoscopic image. Background Art
[0003] An endoscope is a device used to capture images of the digestive tract, such as those inside the human body. Because a single endoscopic examination typically produces tens of thousands of images, outputting all of these images in high definition would consume excessive bandwidth and require long transmission times. Therefore, these images are typically compressed to a lower resolution. Furthermore, the relatively poor imaging environment within the digestive tract makes it difficult to recognize these low-resolution images, hindering doctors' interpretation.
[0004] If the resolution of these images is improved through the existing deep neural network-based image quality enhancement model, doctors will have to spend a lot of waiting time due to the large amount of data and slow model operation speed. Moreover, since the existing image quality enhancement model is not specifically designed for endoscopic images, it cannot improve doctors' reading efficiency.
[0005] Based on this, it is necessary to provide a method that can greatly speed up the output of endoscopic images and effectively improve the resolution of images, so as to improve the efficiency of doctors' reading of images. Summary of the Invention
[0006] In order to solve at least one of the above-mentioned problems in the prior art, an object of the present invention is to provide a method and system for enhancing the image quality of endoscopic images with fast output speed and good output quality.
[0007] To achieve the above-mentioned object of the invention, one embodiment of the present invention provides a method for enhancing the image quality of an endoscopic image, comprising the following steps:
[0008] Obtaining a complex image quality enhancement model, a lightweight image quality enhancement model, an original endoscopic image group, and a low-definition image group, wherein the lightweight image quality enhancement model has fewer intermediate blocks than the complex image quality enhancement model, the original endoscopic image group includes multiple images of the digestive tract acquired by an endoscope, and each low-definition image in the low-definition image group has a corresponding high-definition image in the original endoscopic image group;
[0009] Inputting the low-definition image group into a complex image quality enhancement model and a lightweight image quality enhancement model, wherein the complex image quality enhancement model outputs a first image quality enhanced endoscopic image group, and the lightweight image quality enhancement model outputs a second image quality enhanced endoscopic image group;
[0010] The images of the low-definition image group are subjected to an edge-preserving filtering algorithm to generate a blurred endoscopic image group;
[0011] Selecting n intermediate blocks in the complex image quality enhancement model and the lightweight image quality enhancement model in order from shallow to deep, corresponding intermediate features output by the n intermediate blocks of the complex image quality enhancement model and the lightweight image quality enhancement model in order, and calculating the intermediate feature loss between the n intermediate features of the complex image quality enhancement model and the corresponding intermediate features of the lightweight image quality enhancement model;
[0012] calculating a contrast loss among the first image quality enhanced endoscopic image group, the second image quality enhanced endoscopic image group, and the blurred endoscopic image group;
[0013] calculating reconstruction losses among the first image quality enhanced endoscopic image group, the second image quality enhanced endoscopic image group, and the original endoscopic image group;
[0014] Calculating a total loss based on the intermediate feature loss, the contrast loss, and the reconstruction loss;
[0015] Based on the total loss and the complex image quality enhancement model, the lightweight image quality enhancement model is trained.
[0016] Compared with the prior art, the present invention has the following beneficial effects: the image quality enhancement method of the endoscopic image first improves the clarity of the endoscopic image through a complex image quality enhancement model, and then by combining the intermediate feature loss, contrast loss and reconstruction loss, on the one hand, the computing power of the complex image quality enhancement model is transplanted into the lightweight image quality enhancement model, and on the other hand, the processing capabilities of the complex image quality enhancement model and the lightweight image quality enhancement model for improving the resolution of the endoscopic image are simultaneously improved, so that the lightweight image quality enhancement model finally trained has a strong processing capability for improving the resolution of the endoscopic image, so that the running speed of the model is greatly improved, thereby reducing the waiting time for processing tens of thousands of endoscopic images from several hours to less than one hour, greatly improving the efficiency of doctors in reading films. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] FIG1 is a flow chart of a method for enhancing the image quality of an endoscopic image according to an embodiment of the present invention;
[0018] FIG2 is a flowchart of the specific steps of step S40 in FIG1 ;
[0019] 3 is a network structure framework diagram of an endoscopic image quality enhancement model according to an embodiment of the present invention;
[0020] FIG4 is a structural block diagram of an endoscopic image quality enhancement system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0021] One embodiment of the present invention provides a method and system for enhancing the image quality of endoscopic images with fast output speed and high quality. Endoscopic images can be captured by a camera module on a capsule-shaped endoscope moving within the digestive tract during an endoscopic examination. These images are then transmitted to a device outside the human body for storage, analysis, and interpretation by a physician.
[0022] The following describes an endoscopic image quality enhancement method provided by an embodiment of the present invention in conjunction with Figures 1 to 3. Although this application provides the method operation steps shown in the following embodiments or flowcharts, based on routine or no creative work, the execution order of the steps in the method that are not logically causally related is not limited to the execution order provided in the embodiments of this application. For example, the acquisition order of steps S20, S30, and S40 below can be adjusted arbitrarily, without distinguishing the order in time.
[0023] Specifically, the method for enhancing the image quality of an endoscopic image in this embodiment includes the following steps:
[0024] Step S10: Obtain the complex image quality enhancement model 10, the lightweight image quality enhancement model 20, and the original endoscopic image group O GT And low-definition image group LR.
[0025] Here we explain it from two aspects: image group and model.
[0026] (1) Image group
[0027] Original endoscopic image group O GT The low-resolution image group LR is a data set of pre-made and matched endoscopic images, and the original endoscopic image group O GT The low-resolution image group LR includes a plurality of digestive tract images acquired by an endoscope, wherein each low-resolution image in the original endoscopic image group O is GT There are corresponding high-definition images in the , low-definition image group LR is composed of the original endoscopic image group O GT The degraded image is obtained to form a corresponding low-definition and high-definition image pair.
[0028] Furthermore, in this embodiment, the unique image formation process of the endoscopic image is simulated. Each low-definition image of the low-definition image group LR is formed by the image degradation unit in the following degradation manner: GT The image in is blurred, noise injected and compressed to obtain the corresponding low-definition image generated by the high-definition image.
[0029] Specifically, the image degradation unit includes an image blurring unit, a noise injection unit, and an image compression unit, wherein:
[0030] The image blur unit is configured to simulate lens blur. The camera lens blur kernel is calibrated by calculating a point spread function, and a Gaussian blur kernel is used to simulate the blurring caused by lens shake. The noise injection unit is configured to simulate signal noise. The camera dark current noise and signal Gaussian noise are calibrated by shooting in a darkroom. The image compression unit is configured to perform image compression, using the cv2.encode and cv2.decode functions in the OpenCV library to implement JPEG (Joint Photographic Experts Group) compression.
[0031] (2) Model
[0032] The complex image quality enhancement model 10 can be a pre-trained image quality enhancement model, which is used to improve the image quality of the endoscopic image. First, the generator network is trained using the L1 loss, and the original endoscopic image set O is obtained using the above GT The initialization model is obtained by LR training with a low-definition image group, and then the GAN (Generative Adversarial Network) network is used for adversarial training of the initialization model in the pre-training stage. The L1 loss, perceptual loss and GAN loss can be combined for joint training to obtain a complex image quality enhancement model with excellent performance10, which is a complex large model for improving image quality.
[0033] In this embodiment, the main structure of the network of the complex image quality enhancement model 10 is based on the dense convolutional network RRDB (Residual in Residual Dense Block), the number of intermediate feature channels is 64, and the number of intermediate blocks is 23.
[0034] In order to improve the running speed of the image quality enhancement model on the industrial computer, this embodiment compresses the complex image quality enhancement model 10 into a lightweight image quality enhancement model 20. The main structure of the lightweight image quality enhancement model 20 is consistent with the network of the complex image quality enhancement model 10. The number of intermediate blocks of the lightweight image quality enhancement model 20 is less than that of the complex image quality enhancement model 10. Different numbers of intermediate blocks can be tried through experiments to determine the optimal solution. In this embodiment, the main structure of the lightweight image quality enhancement model 20 is an RRDB network with 64 intermediate feature channels and 11 intermediate blocks. Since the number of intermediate blocks of the lightweight image quality enhancement model 20 is 11, the network parameter amount of the lightweight image quality enhancement model 20 is reduced to about half of the network parameter amount of the complex image quality enhancement model 10.
[0035] Step S20: input the low-definition image group LR into the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20, and the complex image quality enhancement model 10 outputs a first image quality enhanced endoscopic image group O T The lightweight image quality enhancement model 20 outputs a second image quality enhanced endoscopic image group O S .
[0036] The structural framework diagram of the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20, as shown in Figure 3, both include multiple intermediate blocks 11, upsampling blocks 12 and convolutional layers 13. The low-definition image is input into the intermediate block 11 of the first layer, and then passes through the deeper intermediate blocks 11 in sequence, and then passes through the upsampling block 12, and finally outputs the high-definition image from the convolutional layer 13.
[0037] The low-definition image group LR passes through the complex image quality enhancement model 10, and the convolution layer 13 outputs the first image quality enhanced endoscopic image group O T , and several intermediate features output by the intermediate blocks 11; the low-definition image group LR passes through the lightweight image quality enhancement model 20, and the convolution layer 13 outputs the second image quality enhanced endoscopic image group O S , and several intermediate features output by the intermediate blocks 11.
[0038] Step S30: The image of the low-resolution image group LR is subjected to an edge-preserving filtering algorithm to generate a blurred endoscopic image group O. Neg .
[0039] Blurred endoscopic image group O Neg It can be used as a negative sample to improve the lower bound of the model optimization space. The lightweight image quality enhancement model 20 needs to learn to convert the input low-resolution image into the output high-resolution image and should be able to distinguish the difference between the blurred image and the high-resolution image. NegThis helps the lightweight image quality enhancement model 20 to better capture features such as details and textures in the image, thereby improving the performance and robustness of the image quality enhancement model and better solving the image quality enhancement problem.
[0040] In this embodiment, the negative samples generated by the low-resolution image set LR can constrain the search space of the lightweight image quality enhancement model 20, helping the lightweight image quality enhancement model 20 converge to a global optimum. Because the negative samples generated by the relevant low-resolution image set LR are closer to the output results of the lightweight image quality enhancement model 20 and their distance in the latent feature space is closer, the negative samples are of great reference value and help the lightweight image quality enhancement model 20 learn useful information.
[0041] Blurred endoscopic image group O Neg As the negative label of the desired clear image, it needs to include the overall structural framework of the desired image, ignoring high-frequency information (detailed texture). Here, step S30 is further explained in conjunction with Figures 3 and 4. Step S30 is implemented by the negative sample generation device 130. The negative sample generation device 130 includes a blur block 15. Various edge-preserving filter functions can serve as the main function of the blur block 15.
[0042] Step S30 specifically includes: Step S31: upsampling the image of the low-definition image group LR to an image having the same quality as the second image quality enhanced endoscopic image group O S Step S32: The up-sampled low-definition image group LR is blurred to generate a blurred endoscopic image group O Neg .
[0043] Step S32 may further include:
[0044] Step S321: downsampling the upsampled images of the low-resolution image group LR to low-resolution images;
[0045] Step S322: upsampling the downsampled low-resolution image set LR to a level that is consistent with the second image quality enhanced endoscopic image set O. S The resolution is consistent.
[0046] Blurred processing can be performed by downsampling and then upsampling to obtain the blurred endoscopic image group O Neg , sampling can be done using the bicubic interpolation (BiCubic) method. Specifically, the blur block 15 consists of one BiCubic downsampling and one BiCubic upsampling. The generated blurred image preserves the low-frequency information of the low-definition image group LR, that is, the overall frame structure information, which is more valuable as a negative sample than other irrelevant images. Calculation of blurred endoscopic image group O Neg The formula is as follows: Neg=UP(Down(I)). Down and Up represent BiCubic downsampling and BiCubic upsampling, respectively.
[0047] Furthermore, the original endoscopic image group O GT , the first image quality enhanced endoscopic image group O T , the second image quality enhanced endoscopic image group O S , and the blurred endoscopic image group O Neg The images have the same resolution.
[0048] The contrast loss and reconstruction loss mentioned below are both calculated based on images of the same resolution.
[0049] Step S40: Calculating the intermediate feature loss between the n intermediate features of the complex image quality enhancement model 10 and the corresponding intermediate features of the lightweight image quality enhancement model 20 .
[0050] Intermediate feature loss is achieved by extracting features in the intermediate layers of the neural network. These features are used as supervisory signals and compared with the reconstructed image to calculate the reconstruction error. This embodiment uses the intermediate features of the complex image quality enhancement model 10 to supervise the intermediate features of the lightweight image quality enhancement model 20. The network structure of the intermediate feature comparison output by the intermediate n intermediate blocks 11 greatly broadens the expression range of the image quality enhancement method for endoscopic images. During the training process of the network, the intermediate layer feature loss is added to the loss function of the network to encourage the network to generate more consistent feature representations. The intermediate feature loss uses the intermediate layer features as supervisory signals to encourage the model to maintain consistency in the features generated in the intermediate layers, thereby improving the reconstruction quality and generalization ability of the model.
[0051] The step S40 may be specifically shown in FIG. 2 , and includes the following step S41 : selecting n intermediate blocks 11 in the order from shallow to deep in both the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20 .
[0052] According to the principle that shallow features supervise shallow features and deep features supervise deep features, n intermediate blocks 11 are selected from the intermediate blocks 11 of the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20 in order from shallow to deep, wherein the n intermediate blocks 11 of the complex image quality enhancement model 10 and the n intermediate blocks 11 of the lightweight image quality enhancement model 20 are divided into n components of basically equal number according to the number of their respective intermediate blocks 11, and then the intermediate blocks 11 are respectively selected from the n components at basically equal step sizes.
[0053] First, the n components that are substantially equal in number can be understood as follows: (1) If the number of intermediate blocks of the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20 is a multiple of n, then they can be equally divided into n parts. For example, if n is 3, and the number of intermediate blocks of the complex image quality enhancement model 10 is 24, then every 8 blocks are divided into a part; if the number of intermediate blocks of the lightweight image quality enhancement model 20 is 12, then every 4 blocks are divided into a part. (2) If at least one of the number of intermediate blocks of the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20 is not a multiple of 3, for example, if the number of intermediate blocks of the complex image quality enhancement model 10 is 23, then two of the parts are 8 blocks and one is 7 blocks; if the number of intermediate blocks of the lightweight image quality enhancement model 20 is 11, then two of the parts are 4 blocks and one is 3 blocks, and so on.
[0054] Secondly, the middle blocks 11 selected from the three components with substantially equal step sizes can be understood as follows:
[0055] For the above-mentioned if (1), since the model is divided into three equal parts, each part is equal, the step length can be completely consistent. For example, in the example where the number of intermediate blocks in the complex image quality enhancement model 10 is 24, then every 8 blocks are divided into one part, and the step length can also be 8, that is, one intermediate block 11 is selected for every 8 intermediate blocks 11; in the example where the number of intermediate blocks in the lightweight image quality enhancement model 20 is 12, then every 4 blocks are divided into one part, and the step length can also be 4, that is, one intermediate block 11 is selected for every 4 intermediate blocks 11.
[0056] For the above-mentioned if (2), since the number of intermediate blocks is not a multiple of 3, the step size will also change. For example, in the example where the number of intermediate blocks of the complex image quality enhancement model 10 is 23 blocks, two of the blocks are 8 blocks and one of the blocks is 7 blocks, in the 8-block part, the step size is 8, and in the 7-block part, the step size is 7; for another example, in the example where the number of intermediate blocks of the lightweight image quality enhancement model 20 is 11 blocks, two of the blocks are 4 blocks and one of the blocks is 3 blocks, in the 4-block part, the step size is 4, and in the 3-block part, the step size is 3, and so on.
[0057] Specifically, in this embodiment, the complex image quality enhancement model 10 has 23 intermediate blocks, which cannot be divided into three equal parts. Therefore, the complex image quality enhancement model 10 is divided into three parts: 8, 7, and 8. The seventh intermediate block 11 in the first part of the first 8 blocks is selected; the second block is located in the second part of the 7 blocks, and the step size is 7, that is, the 14th intermediate block 11 in the second part is selected; the third block is located in the third part of the second 8 blocks, and the step size is 8, that is, the 22nd intermediate block 11 in the third part is selected.
[0058] The lightweight image quality enhancement model 20 has 11 intermediate blocks, which cannot be divided into three equal parts. Therefore, the lightweight image quality enhancement model 20 is divided into three parts: 4, 4, and 3. The third intermediate block 11 in the first part of the first four blocks is selected. The second block is located in the second part of the second four blocks, and the step size is selected as 4, that is, the seventh intermediate block 11 in the second part is selected. The third block is located in the third part of the three blocks, and the step size is selected as 3, that is, the tenth intermediate block 11 in the third part is selected.
[0059] Step S42: The intermediate features output by the n intermediate blocks 11 of the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20 are matched one by one in sequence.
[0060] According to the order of the respective intermediate blocks 11, the intermediate features output by the intermediate blocks 11 selected from the complex image quality enhancement model 10 are matched one by one with the intermediate features output by the intermediate blocks 11 selected from the lightweight image quality enhancement model 20. Still taking the above example as an example, three sets of corresponding intermediate features will be generated in sequence. Accordingly, the intermediate features output by the 7th intermediate block 11 of the complex image quality enhancement model 10 supervise the intermediate features output by the 3rd intermediate block 11 of the lightweight image quality enhancement model 20, forming a 7-3 feature pair; the intermediate features output by the 14th intermediate block 11 of the complex image quality enhancement model 10 supervise the intermediate features output by the 7th intermediate block 11 of the lightweight image quality enhancement model 20, forming a 14-7 feature pair; the intermediate features output by the 22nd intermediate block 11 of the complex image quality enhancement model 10 supervise the intermediate features output by the 10th intermediate block 11 of the lightweight image quality enhancement model 20, forming a 22-10 feature pair.
[0061] Step S43: performing feature transformation on the intermediate features of the complex image quality enhancement model 10 and the intermediate features of the lightweight image quality enhancement model 20, and transforming them into feature spaces of the same dimension.
[0062] Feature transformation can be a convolution operation, pooling operation, or other transformations. In addition, additional transformations such as local response normalization (LRN), upsampling or downsampling can also be performed. By transforming the features of the middle layer of the network, the expressive power of the model is enhanced and the information loss during the feature transformation process is minimized.
[0063] In this embodiment, the feature transformation of the complex image quality enhancement model 10 and the feature transformation of the lightweight image quality enhancement model 20 both use a convolutional layer with a kernel size of 1x1, and the number of output channels is the same as the number of intermediate features of the lightweight image quality enhancement model 20.
[0064] Step S44: Calculate the distance loss between the n transformed intermediate features of the complex image quality enhancement model 10 and the corresponding intermediate features of the lightweight image quality enhancement model 20.
[0065] Calculate the distance loss between the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20. The distance function adopts the L1 loss function, that is, calculates the sum of the absolute values of the differences between the predicted value and the true value. In the image quality restoration task, the L1 loss can promote the network to restore a clearer image and be more robust to outliers, that is, it will not be affected by outliers. Taking the above text as an example, the L1 losses of the 7-3, 14-7, and 22-10 feature pairs corresponding to the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20 are calculated here in turn. The calculation formula of the L1 loss is as follows: , (Formula 1), where I1 is one image, I2 is the other image, N is the number of pixels on the horizontal axis of the image, and M is the number of pixels on the vertical axis of the image.
[0066] Step S45: summing and averaging the n distance losses to obtain the intermediate feature loss.
[0067] Intermediate feature loss L fea The calculation formula is as follows: , (Formula 2), where the calculation formula of L1 refers to Formula 1, (T7, S3) is the above 7-3 feature pair, (T 14 , S7) is the above 14-7 feature pair, (T 22 , S 10 ) is the 22-10 feature pair mentioned above.
[0068] Step S50: Calculate the first image quality enhanced endoscopic image group O T , the second image quality enhanced endoscopic image group O S and the blurred endoscopic image group O Neg The contrast loss between them.
[0069] Step S50 ensures that the output of the lightweight image quality enhancement model 20 is as close as possible to the output of the complex image quality enhancement model 10, and that the output of the lightweight image quality enhancement model 20 is far away from negative samples, thereby improving the lower bound of the lightweight image quality enhancement model 20, reducing the learning space of the lightweight image quality enhancement model 20, and promoting the optimization of the lightweight image quality enhancement model 20 towards the output of the complex image quality enhancement model 10. By bringing similar samples as close as possible and dissimilar samples as far apart as possible, better feature representations are learned, which can effectively alleviate problems such as sample imbalance and category noise, and improve the robustness and generalization ability of the model.
[0070] Here, the contrast loss can be calculated by calculating the distance. By designing a suitable distance evaluation function, the distance between the output vector of the complex image quality enhancement model 10 and the output vector of the lightweight image quality enhancement model 20 can be reduced as much as possible, so that the output vector of the complex image quality enhancement model 10 and the output vector of the lightweight image quality enhancement model 20 are as similar as possible.
[0071] Here, a suitable latent feature space is selected as the low-dimensional space for the input data. The latent feature space represents the raw data in a form that is more separable and interpretable, allowing the output results of the complex image quality enhancement model 10, the output results of the lightweight image quality enhancement model 20, and negative samples to be measured relative to each other in the space. The latent feature space is learned by the machine learning model through operations such as dimensionality reduction or feature extraction on the raw data. In this low-dimensional space, the input data is typically represented as a set of continuous vectors, which can be used for further data processing, classification, clustering, and other tasks.
[0072] This embodiment uses the VGG19 network as the mapping function of the latent feature space. The VGG19 network has good feature extraction capabilities and can extract high-dimensional spatial information of input features.
[0073] The contrast loss L CL The calculation formula is: , (Formula 3)
[0074] Where: N represents the number of samples, M represents the number of features output by the mapping function of the latent feature space, λ j represents the weight of the jth feature, φ represents the mapping function of the latent feature space, and K represents the fuzzy endoscopic image group O Neg The number of images; T , O S , O Neg respectively represent the first image quality enhanced endoscopic image group O T , the second image quality enhanced endoscopic image group O S and the blurred endoscopic image group O Neg Image, blurred endoscope image group O Neg Refer to step S30 above; d represents the distance function, where the L1 loss function is used, and the specific calculation method uses the above formula 1.
[0075] According to formula 3, when the output of the lightweight image quality enhancement model 20 is closer to the output of the complex image quality enhancement model 10 and farther away from the negative sample, the contrast loss L CL The smaller the value, the greater the contrast loss L CL The bigger.
[0076] Step S60: Calculate the first image quality enhanced endoscopic image group O T, the second image quality enhanced endoscopic image group O S and the original endoscopic image group O GT The reconstruction loss between .
[0077] The reconstruction loss is used to prevent the output of the lightweight image quality enhancement model 20 from generating a large color shift. The reconstruction loss L rec The calculation formula is: , (Formula 4), where N represents the original endoscopic image group O GT The number of images, O T , O S , O Neg respectively represent the first image quality enhanced endoscopic image group O T , the second image quality enhanced endoscopic image group O s and the original endoscopic image group O GT The image is , d represents the distance function, the L1 loss function is used here, and the specific calculation method is the formula 1 above.
[0078] Step S70: Calculate the total loss based on the intermediate feature loss, the contrast loss and the reconstruction loss.
[0079] The calculation formula of the total loss L is: L=L rec +αL fea +βL CL , where L rec is the reconstruction loss, and its algorithm refers to formula 4; L fea is the intermediate feature loss, and its algorithm refers to formula 2; L CL is the contrast loss, and its algorithm refers to Formula 3; α and β represent the weights of the intermediate feature loss and the contrast loss, respectively, and can be dynamically adjusted according to experiments.
[0080] In addition, the L1 loss function (Formula 1) used above to calculate the distance d can also be replaced by other loss functions, such as the L2 loss function.
[0081] Step S80: Based on the total loss and the complex image quality enhancement model 10, the lightweight image quality enhancement model 20 is trained.
[0082] During training, the backpropagation algorithm is used to calculate the gradient of the total loss with respect to the lightweight image quality enhancement model 20. Then, optimization algorithms such as gradient descent are used to update the parameters of the lightweight image quality enhancement model 20. These steps are repeated, iteratively updating the parameters of the lightweight image quality enhancement model 20 until the lightweight image quality enhancement model 20 converges. Only the gradient of the lightweight image quality enhancement model 20 is updated; the parameters of the complex image quality enhancement model 10 are not included in the training. The lightweight image quality enhancement model 20 is trained according to the framework in Figure 3 to obtain a complete model.
[0083] Step S80 may specifically include the following steps: freezing the parameters of the complex image quality enhancement model 10; changing the loss function and fine-tuning the lightweight image quality enhancement model 20 to balance the knowledge of the complex image quality enhancement model 10 and its own capabilities; adjusting the learning rate to balance the learning speed and the convergence quality of the model; increasing training data to improve the generalization ability of the model; performing data enhancement to improve the robustness and generalization ability of the model; and iterative fine-tuning until the best result is obtained.
[0084] After experimental verification, the low-definition image group LR is processed by the lightweight image quality enhancement model 20, and the output second image quality enhanced endoscopic image group O S With the original endoscopic image group O GT There is no obvious visual difference between the images. The texture details of the stomach wall image are similar to those of the original endoscopic image group O. GT The results are basically the same, and the quality of the generated image is not significantly reduced. In addition, the lightweight image quality enhancement model 20 is small in size and runs much faster than the complex image quality enhancement model 10. This shows that the lightweight image quality enhancement model 20 has a fast output speed and good image quality improvement effect.
[0085] Compared with the prior art, this embodiment has the following beneficial effects:
[0086] (1) The above-mentioned endoscopic image quality enhancement method transplants the computing power of the complex image quality enhancement model 10 into the lightweight image quality enhancement model 20 without increasing the additional computing cost, and uses the complex image quality enhancement model 10 to improve the performance of the lightweight image quality enhancement model 20.
[0087] (2) The processing capabilities of the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20 for improving the resolution of endoscopic images are improved, so that the lightweight image quality enhancement model 20 finally trained has a stronger processing capability for improving the resolution of endoscopic images.
[0088] (3) The running speed of the model has been greatly improved, thereby reducing the waiting time for processing tens of thousands of endoscopic images from several hours to less than one hour, greatly improving the efficiency of doctors' reading of images.
[0089] In one embodiment, a system 100 for enhancing the image quality of an endoscopic image is provided, as shown in FIG4 . The system 100 for enhancing the image quality of an endoscopic image includes:
[0090] The acquisition device 110 is configured to acquire the complex image quality enhancement model 10, the lightweight image quality enhancement model 20, the original endoscopic image group O GTand a low-definition image group LR, wherein the number of intermediate blocks of the lightweight image quality enhancement model 20 is less than that of the complex image quality enhancement model 10, and the original endoscopic image group O GT The low-resolution image group LR includes a plurality of digestive tract images acquired by an endoscope, wherein each low-resolution image in the original endoscopic image group O is GT There are corresponding high-definition images in;
[0091] The output device 120 is configured to input the low-definition image group LR into the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20, and the complex image quality enhancement model 10 outputs the first image quality enhanced endoscopic image group O T The lightweight image quality enhancement model 20 outputs a second image quality enhanced endoscopic image group O S ;
[0092] The negative sample generating device 130 is configured to generate a blurred endoscopic image group O by using an edge-preserving filtering algorithm on the image of the low-resolution image group LR. Neg ;
[0093] The intermediate feature loss calculation device 140 is configured to select n intermediate blocks 11 from the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20 in order from shallow to deep, correspond the intermediate features output by the n intermediate blocks 11 of the complex image quality enhancement model 10 and the lightweight image quality enhancement model 20 in order, and calculate the intermediate feature loss between the n intermediate features of the complex image quality enhancement model 10 and the corresponding intermediate features of the lightweight image quality enhancement model 20;
[0094] The contrast loss calculation device 150 is configured to calculate the first image quality enhanced endoscopic image group O T , the second image quality enhanced endoscopic image group O S and the blurred endoscopic image group O Neg The contrast loss between
[0095] The reconstruction loss calculation device 160 is configured to calculate the first image quality enhanced endoscopic image group O T , the second image quality enhanced endoscopic image group O S and the original endoscopic image group O GT reconstruction losses between;
[0096] a total loss calculation device 170 configured to calculate a total loss based on the intermediate feature loss, the contrast loss, and the reconstruction loss;
[0097] The training device 180 is configured to train the lightweight image quality enhancement model 20 based on the total loss and the complex image quality enhancement model 10.
[0098] In addition, in one embodiment, the negative sample generating device 130 includes an upsampling block 14 and a blurring block 15, wherein the upsampling block 14 is configured to upsample the image of the low-definition image group LR to an image having the same resolution as the second image quality enhanced endoscopic image group O. S The blur block 15 is configured to perform LR blurring on the upsampled low-definition image group to generate a blurred endoscopic image group O. Neg .
[0099] It should be noted that for details not disclosed in the endoscopic image quality enhancement system according to the embodiment of the present invention, please refer to the details disclosed in the endoscopic image quality enhancement method according to the embodiment of the present invention.
[0100] The endoscopic image quality enhancement system may also include a computing device such as a computer, a laptop, a PDA, and a cloud server, as well as a processor, a memory, and a computer program stored in the memory and executable by the processor, such as the aforementioned endoscopic image quality enhancement method program. When the processor executes the computer program, the steps of each of the aforementioned endoscopic image quality enhancement method embodiments are implemented, such as the steps shown in FIG1 or 2.
[0101] In addition, the present invention also proposes an electronic device, which includes a memory and a processor. When the processor executes the computer program, it can implement the steps in the above-mentioned method for enhancing the image quality of endoscopic images, that is, implement the steps in any one of the technical solutions in the above-mentioned method for enhancing the image quality of endoscopic images.
[0102] The electronic device may be a part of an image quality enhancement system integrated into an endoscopic image, or a local terminal device, or a part of a cloud server.
[0103] The processor can be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor is the control center of the endoscopic image quality enhancement system and connects the various parts of the entire endoscopic image quality enhancement system using various interfaces and circuits.
[0104] The memory can be used to store the computer programs and / or modules. The processor implements the various functions of the endoscopic image quality enhancement system by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc. In addition, the memory can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.
[0105] Exemplarily, a computer program may be divided into one or more modules / units, which are stored in a memory and executed by a processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the endoscopic image quality enhancement system.
[0106] Furthermore, an embodiment of the present invention provides a readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-mentioned method for enhancing the image quality of endoscopic images, and implement the steps in any one of the technical solutions in the above-mentioned method for enhancing the image quality of endoscopic images.
[0107] If the module integrated into the endoscopic image quality enhancement method is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The present invention can also implement all or part of the processes in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments.
[0108] The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording medium, disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium.
[0109] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for enhancing the image quality of an endoscopic image, characterized in that: The steps include: Acquire a complex image quality enhancement model, a lightweight image quality enhancement model, an original endoscopic image group, and a low-definition image group, wherein the number of intermediate blocks of the lightweight image quality enhancement model is less than that of the complex image quality enhancement model, the original endoscopic image group includes a plurality of digestive tract images acquired by an endoscope, and each low-definition image of the low-definition image group has a corresponding high-definition image in the original endoscopic image group; Inputting the low-definition image group into a complex image quality enhancement model and a lightweight image quality enhancement model, wherein the complex image quality enhancement model outputs a first image quality enhanced endoscopic image group, and the lightweight image quality enhancement model outputs a second image quality enhanced endoscopic image group; The images of the low-definition image group are subjected to an edge-preserving filtering algorithm to generate a blurred endoscopic image group; Selecting n intermediate blocks in the complex image quality enhancement model and the lightweight image quality enhancement model in order from shallow to deep, making the intermediate features output by the n intermediate blocks of the complex image quality enhancement model and the lightweight image quality enhancement model correspond one to one in order, and calculating the intermediate feature loss between the n intermediate features of the complex image quality enhancement model and the corresponding intermediate features of the lightweight image quality enhancement model; calculating a contrast loss among the first image quality enhanced endoscopic image group, the second image quality enhanced endoscopic image group, and the blurred endoscopic image group; calculating a reconstruction loss between the first image quality enhanced endoscopic image group, the second image quality enhanced endoscopic image group, and the original endoscopic image group; Calculating a total loss according to the intermediate feature loss, the contrast loss, and the reconstruction loss; Based on the total loss and the complex image quality enhancement model, the lightweight image quality enhancement model is trained.
2. The method for enhancing the image quality of an endoscopic image according to claim 1, characterized in that: Each low-definition image in the low-definition image group is obtained by blurring, injecting noise and compressing the images in the original endoscopic image group.
3. The method for enhancing the image quality of an endoscopic image according to claim 1, characterized in that: The step of selecting n intermediate blocks in the complex image quality enhancement model and the lightweight image quality enhancement model in order from shallow to deep, and corresponding the intermediate features output by the n intermediate blocks of the complex image quality enhancement model and the lightweight image quality enhancement model in order comprises: n intermediate blocks are selected from the intermediate blocks of the complex image quality enhancement model and the lightweight image quality enhancement model in order from shallow to deep, wherein the n intermediate blocks of the complex image quality enhancement model and the n intermediate blocks of the lightweight image quality enhancement model are divided into n components of substantially equal number according to the number of their respective intermediate blocks, and then the intermediate blocks are respectively selected from the n components in substantially equal step sizes; According to the order of the respective intermediate blocks, the intermediate features output by the intermediate blocks selected in the complex image quality enhancement model are matched one by one with the intermediate features output by the intermediate blocks selected in the lightweight image quality enhancement model, and n groups of corresponding intermediate features are generated in turn.
4. The method for enhancing the image quality of an endoscopic image according to claim 1, characterized in that: The step of calculating the intermediate feature loss between the n intermediate features of the complex image quality enhancement model and the corresponding intermediate features of the lightweight image quality enhancement model comprises: Performing feature transformation on the intermediate features of the complex image quality enhancement model and the intermediate features of the lightweight image quality enhancement model, and transforming them into feature spaces of the same dimension; Calculating the distance loss between the n transformed intermediate features of the complex image quality enhancement model and the corresponding intermediate features of the lightweight image quality enhancement model; The intermediate feature loss is obtained by adding and averaging the n distance losses.
5. The method for enhancing the image quality of an endoscopic image according to claim 1, characterized in that: The original endoscopic image group, the first image quality enhanced endoscopic image group, the second image quality enhanced endoscopic image group, and the blurred endoscopic image group all have the same resolution.
6. The method for enhancing the image quality of an endoscopic image according to claim 5, characterized in that: The step of generating a blurred endoscopic image group from the images of the low-definition image group by an edge-preserving filtering algorithm comprises: Upsampling the images of the low-definition image group to a resolution consistent with that of the second image quality enhanced endoscopic image group; The upsampled low-definition image group is blurred to generate a blurred endoscopic image group.
7. The method for enhancing the image quality of an endoscopic image according to claim 5, characterized in that: The step of calculating the contrast loss between the first image quality enhanced endoscopic image group, the second image quality enhanced endoscopic image group and the blurred endoscopic image group comprises: The VGG19 network is used as the mapping function of the latent feature space; The contrast loss L CL The calculation formula is: , Where: N represents the number of samples, M represents the number of features output by the mapping function of the latent feature space, and λ j represents the weight of the jth feature, φ represents the mapping function of the latent feature space, K represents the number of images in the fuzzy endoscopic image group, O T , O S , O Neg They respectively represent images of the first image quality enhanced endoscopic image group, the second image quality enhanced endoscopic image group and the blurred endoscopic image group, and d represents a distance function.
8. The method for enhancing the image quality of an endoscopic image according to claim 5, characterized in that: The step of calculating the reconstruction loss between the first image quality enhanced endoscopic image group, the second image quality enhanced endoscopic image group and the original endoscopic image group comprises: The reconstruction loss L rec The calculation formula is: , Wherein, N represents the number of images in the original endoscopic image group, O T , O S , O Neg They respectively represent images of the first image quality enhanced endoscopic image group, the second image quality enhanced endoscopic image group and the original endoscopic image group, and d represents a distance function.
9. The method for enhancing the image quality of an endoscopic image according to claim 5, characterized in that: The step of calculating the total loss according to the intermediate feature loss, the contrast loss and the reconstruction loss comprises: The total loss is calculated as follows: L = L rec +αL fea +βL CL , Among them, L rec is the reconstruction loss, L fea is the intermediate feature loss, L CL is the contrast loss, α and β represent the weights of the intermediate feature loss and the contrast loss, respectively.
10. An endoscopic image quality enhancement system, characterized in that: include: An acquisition device is configured to acquire a complex image quality enhancement model, a lightweight image quality enhancement model, an original endoscopic image group, and a low-definition image group, wherein the number of intermediate blocks of the lightweight image quality enhancement model is less than that of the complex image quality enhancement model, the original endoscopic image group includes a plurality of digestive tract images acquired by an endoscope, and each low-definition image of the low-definition image group has a corresponding high-definition image in the original endoscopic image group; an output device configured to input the low-definition image group into a complex image quality enhancement model and a lightweight image quality enhancement model, the complex image quality enhancement model outputs a first image quality enhanced endoscopic image group, and the lightweight image quality enhancement model outputs a second image quality enhanced endoscopic image group; A negative sample generating device is configured to generate a blurred endoscopic image group by using an edge-preserving filtering algorithm on the images of the low-definition image group; The intermediate feature loss calculation device is configured to select n intermediate blocks in the complex image quality enhancement model and the lightweight image quality enhancement model in order from shallow to deep, correspond the intermediate features output by the n intermediate blocks of the complex image quality enhancement model and the lightweight image quality enhancement model in order one by one, and calculate the intermediate feature loss between the n intermediate features of the complex image quality enhancement model and the corresponding intermediate features of the lightweight image quality enhancement model; a contrast loss calculation device configured to calculate a contrast loss between the first image quality enhanced endoscopic image group, the second image quality enhanced endoscopic image group, and the blurred endoscopic image group; a reconstruction loss calculation device configured to calculate a reconstruction loss between the first image quality enhanced endoscopic image group, the second image quality enhanced endoscopic image group and the original endoscopic image group; A total loss calculation device, configured to calculate a total loss according to the intermediate feature loss, the contrast loss and the reconstruction loss; The training device is configured to train the lightweight image quality enhancement model based on the total loss and the complex image quality enhancement model.
11. An electronic device, characterized in that: include: Memory, which stores computer programs; The processor can implement the steps of the method for enhancing the image quality of an endoscopic image as claimed in claim 1 when executing the computer program.
12. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps in the method for enhancing the image quality of an endoscopic image described in claim 1 can be implemented.
Citation Information
Patent Citations
Image quality enhancement method and device, electronic equipment and readable storage medium
CN111325692A
Image enhancement model training method, image enhancement method and related device
CN112419219A
Lightweight image defogging method based on cross-stage local connection
CN115641271A
Image enhancement method and device, electronic equipment and storage medium
CN115953317A
Method and device for enhancing image quality of endoscope image
CN116188346A
