Endoscope image dark region enhancement method based on deep learning and self-group iteration upgrading device

Abstract

The invention discloses a deep learning-based endoscope image dark region enhancement method and a self-group iteration upgrading device. The method comprises the following steps of 1, collecting biological tissue images of an endoscope under a low illumination condition and a normal illumination condition; 2, taking the biological tissue images of the endoscope under the low illumination condition and the normal illumination condition as a training set, and constructing and training a dark region image enhancement network; and 3, inputting an original image obtained by an endoscope into the trained dark region enhancement network, and outputting an image after dark region enhancement. According to the invention, the endoscope image can be processed in real time, the definition of the endoscope image is improved, and the details of the image in the dark area of the endoscope are enhanced. According to the self-group iteration upgrading device, model training iteration upgrading can be carried out in multiple modes, and the performance of the device is continuously improved.

Description

Technical Field

[0001] This invention relates to the field of endoscopic image processing technology, specifically to a method for enhancing dark areas of endoscopic images based on deep learning and a self-group iterative upgrade device. Background Technology

[0002] With the continuous development of medical technology, endoscopic technology has become an inevitable trend in surgical development. Endoscopes can be directly inserted into the human body to screen for lesions and can also provide precise navigation for surgery. Compared with open surgery, endoscopic surgery has advantages such as no abdominal incision, less trauma, and faster recovery, while also improving the diagnostic efficiency of doctors.

[0003] However, due to the characteristics of the human body cavity structure, the endoscope light source needs to be inserted from the outside. And because the endoscope light source is a single warm color, the original endoscope images acquired are often dark, unevenly lit, and have low contrast. However, high-definition and high-contrast images have extremely high clinical value for disease diagnosis, so it is necessary to enhance the endoscope images.

[0004] Histogram enhancement, wavelet analysis enhancement, and homomorphic filtering enhancement are commonly used endoscopic image enhancement algorithms. Histogram enhancement includes global histogram equalization (HPE) and adaptive histogram equalization (AHE). Histogram equalization modifies the grayscale values ​​of pixels in the endoscopic image by changing the image's histogram, effectively enhancing contrast for images with small dynamic ranges. Adaptive histogram equalization enhances contrast by dividing the image into blocks and processing each block, which amplifies noise to some extent. Wavelet analysis-based image enhancement algorithms mainly include wavelet transform high-frequency enhancement and adaptive filtering enhancement. Their basic principle is to decompose the image into different sub-bands, perform relevant processing on each sub-band, and finally use wavelet reconstruction to obtain the image with improved contrast.

[0005] Homomorphic filtering enhancement has a significant effect on image denoising and dehazing. Its basic principle is that the light intensity component of an image is concentrated in the low-frequency part and changes slowly, while the other part is the high-frequency component containing various object information, namely the reflection component. After a series of operations, the enhanced image can be obtained.

[0006] The above-mentioned endoscopic image enhancement methods have the following drawbacks:

[0007] (1) Existing algorithms are too complex and cannot achieve real-time video processing;

[0008] (2) Existing algorithms are not effective. While enhancing dark areas, they also increase the brightness of bright areas, resulting in a decrease in image contrast.

[0009] (3) Existing algorithms produce halo effects where the image brightness changes abruptly, resulting in insufficient clarity of the endoscopic image.

[0010] In recent years, convolutional neural networks (CNNs) have achieved significant breakthroughs in many low-level computer vision tasks, including image super-resolution, deblurring, dehazing, denoising, and image enhancement. Compared to traditional methods, some CNN-based methods have greatly improved the quality of image enhancement. Based on this, this invention provides a deep learning-based method for enhancing dark areas in endoscopic images. Summary of the Invention

[0011] To address the problems of artificially set weights in existing traditional detail enhancement algorithms, which amplify image noise, have high algorithm complexity, slow running speed, and cause image color distortion, this invention provides a deep learning-based method for enhancing dark areas of endoscopic images. The method includes:

[0012] The raw images output from the endoscope are acquired and input into the dark area image enhancement network to obtain the dark area enhanced endoscope images.

[0013] The training method for the dark area image enhancement network is as follows:

[0014] Step S100: Using an endoscope, images of biological tissue are acquired under both low-light and normal-light conditions, and then segmented into... The pixel size is defined as follows: the image acquired by the endoscope in low light is used as the original image, and the image acquired by the endoscope in normal light is used as the target image (label).

[0015] In some preferred embodiments, the original image and the target image (label) are acquired using the following method:

[0016] Step S110: Using an endoscope, images of biological tissue are acquired under low-light conditions and segmented into... Pixel size, used as the original image;

[0017] Step S120: Using an endoscope, images of biological tissue are acquired under normal lighting conditions and segmented into... Pixel size, used as the target image (label);

[0018] Step S200: Using the original endoscope image and the target image (label) as a training set, construct and train a dark area image enhancement network.

[0019] The images obtained by S100 are randomly divided into training, validation, and test sets according to a reasonable ratio (here, 8:1:1) for model training. The training process utilizes a computing platform equipped with an NVIDIA graphics card (Tesla A800 with 80GB). The learning rate of the neural network is set to a reasonable value (here, it can be set to...). The optimizer used is the AdamW optimizer, and the optimizer parameters are set to reasonable values ​​(here, it can be set to...). , ).

[0020] In some preferred embodiments, the training method for the dark area image enhancement network is as follows:

[0021] Step S210, the original image The images are input into a layer decomposition network to obtain low-light training images. and high-light training images ;

[0022] Step S220, use the low-light training images The image is input into the reflector reconstruction network to obtain the first training image. ;

[0023] Step S230, use the high-light training image The image is input into the illumination adjustment network to obtain the second training image. ;

[0024] Step S240: The dark area enhancement image is calculated using the following formula:

[0025]

[0026] In some preferred embodiments, the training method of the dark region enhancement network further includes:

[0027] Step S250: Calculate the mixture loss function of the layer decomposition network. And according to the hybrid loss function The layer decomposition network is modified. The mixture loss function... It can be expressed as

[0028]

[0029] in:

[0030]

[0031]

[0032]

[0033]

[0034] Original image; The reflectance component of the original image; The reflectance component of the original image; and The illumination components are those of the low-light image and the high-light image, respectively; and These are the reflectance components of the low-light and high-light images, respectively.

[0035] Step S260: Calculate the hybrid loss function of the reflectivity reconstruction network. And according to the hybrid loss function The reflector reconstruction network is modified. The hybrid loss function... It can be expressed as

[0036]

[0037] It is an image structure similarity measurement function; and These are the reflectance components of the low-light and high-light images, respectively.

[0038] Step S270: Calculate the hybrid loss function of the illumination adjustment network. And according to the hybrid loss function The illumination adjustment network is modified. The hybrid loss function... It can be expressed as

[0039]

[0040] The reflectance component of the original image; Enhance the reflectance component of the image in dark areas.

[0041] Step S300: Input the original image obtained by the endoscope into the trained dark region enhancement network, and output the image after dark region enhancement.

[0042] In step S400, the original endoscope image is input into the dark area enhancement network self-group iterative upgrade device to perform online correction on the dark area enhancement network.

[0043] The self-group iterative upgrade device includes a single-machine self-iterative upgrade module and a networked group iterative upgrade module. The difference between self-upgrade and group upgrade lies in whether the training information (dataset, dataset size, dataset quality, etc.) originates from a local single machine (self-upgrade) or a networked group (group upgrade). The group iterative upgrade module includes two functions: local group iterative upgrade and cloud group iterative upgrade. The difference between local group upgrade and cloud group upgrade lies in whether the training process is completed locally or in the cloud. The training device can be local, in the cloud, or both. The local group iterative upgrade includes two functions: local group single-machine training iterative upgrade and local group distributed training iterative upgrade. The local group single-machine training iterative upgrade includes two functions: local group single-machine training with shared dataset iterative upgrade (without sharing training weights) and local group single-machine training with shared dataset and weights iterative upgrade.

[0044] The deep learning-based self-iterative upgrade device includes:

[0045] Storage module for collecting raw endoscopic images input to the dark region enhancement network;

[0046] The expert annotation module is used to annotate the raw endoscopic images in the storage module and generate corresponding labels; the expert annotation module includes a camera with light source supplementation.

[0047] The neural network fine-tuning module is used to correct and fine-tune the dark area enhancement network based on the images collected by the storage module;

[0048] The self-iterative upgrade step based on the self-iterative upgrade device is to input the original endoscope image into the dark region enhancement network and train the dark region enhancement network.

[0049] The deep learning-based group iterative upgrade device includes:

[0050] Storage module for collecting raw endoscopic images input to the dark region enhancement network;

[0051] The expert annotation module is used to annotate the raw endoscopic images in the storage module and generate corresponding labels; the expert annotation module includes a camera with light source supplementation.

[0052] The neural network fine-tuning module is used to correct and fine-tune the dark area enhancement network based on the images collected by the storage module;

[0053] The network service module is used for communication and data transmission with the cloud control center.

[0054] The iterative upgrade of the shared local single-machine training dataset based on the group iterative upgrade module includes:

[0055] The training dataset comes from all networked devices;

[0056] Raw endoscopic images shared by several other networked devices are downloaded via a network module and input into the dark region enhancement network of the current local device. The network is trained locally to obtain corrected dark region enhancement network weights for use by the current local device.

[0057] The group-based local single-machine training shared dataset and weight iterative upgrade based on the group iterative upgrade module includes:

[0058] The training dataset comes from all networked devices;

[0059] Prerequisite: All networked devices must use the same training model software;

[0060] Raw endoscopic images from several other networked devices are downloaded to the current local device via a network module and input into the dark region enhancement network of the current local device. The network is then trained to obtain corrected dark region enhancement network weights, which are then updated and shared in real time with all networked devices for their use.

[0061] The group-based local distributed training iterative upgrade based on the group iterative upgrade module includes:

[0062] Training information (dataset size, dataset quality, etc.) comes from all networked devices;

[0063] Prerequisites: Either all networked devices use the same training model software (which requires knowledge of the dataset size for each device), or the quantitative differences in the parameters of the training datasets generated by each device are known (which requires knowledge of the dataset quality for each device).

[0064] The raw endoscopic images from each local device are input into the local device's dark region enhancement network for training, resulting in corrected dark region enhancement network weights. The corrected dark region enhancement network weights from several networked devices are then uploaded to the cloud control center for global reduction (AllReduce). If all networked devices use the same training model software, global reduction is performed based on the size of each device's dataset. If the networked devices use different training model software and the quantitative differences in the dataset parameters of each device are known, global reduction is performed based on the quantitative differences in the quality of each device's dataset. This yields updated dark region enhancement network weights.

[0065] The group cloud-based iterative upgrade based on the group iterative upgrade module includes:

[0066] Prerequisite: The training model software must be identical across all networked devices and cloud-based devices;

[0067] Raw endoscopic images from several networked devices are transmitted to the cloud control center via a network service module; the cloud control center aggregates the received raw endoscopic images and trains the dark region enhancement network; the local network service module downloads and synchronously updates the dark region enhancement network weights for the current device.

[0068] Therefore, compared with the prior art, the method provided by the present invention has the following beneficial effects:

[0069] 1. This invention also has the advantages of adaptively enhancing the brightness of dark areas of an image and suppressing noise;

[0070] 2. This invention can improve image contrast and clarity. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a flowchart of an embodiment of a method for enhancing dark areas of endoscopic images based on deep learning according to the present invention.

[0073] Figure 2 This is a network structure diagram of the dark region image enhancement network in the method for dark region enhancement of endoscopic images based on deep learning according to the present invention.

[0074] Figure 3 This is a schematic diagram of the self-group iterative upgrade device based on deep learning according to the present invention.

[0075] Figure 4 This is a schematic diagram of the self-iterative upgrade process based on deep learning in this invention.

[0076] Figure 5 This is a logical block diagram of the self-iterative upgrade module based on deep learning in this invention.

[0077] Figure 6 This is a schematic diagram of the process of iterative upgrading of a shared dataset based on local single-machine training of a group using deep learning, as described in this invention.

[0078] Figure 7 This is a logical block diagram of the deep learning-based group local single-machine training shared dataset iterative upgrade module of the present invention.

[0079] Figure 8This is a schematic diagram of the process of group local single-machine training and weight iterative upgrade based on deep learning in this invention.

[0080] Figure 9 This is a logical block diagram of the deep learning-based group local single-machine training shared dataset and weight iterative upgrade module of the present invention.

[0081] Figure 10 This is a schematic diagram of the process of iterative upgrade of group local distributed training based on deep learning in this invention.

[0082] Figure 11 This is a logical block diagram of the group-local distributed iterative upgrade module based on deep learning in this invention.

[0083] Figure 12 This is a schematic diagram of the process of the cloud-based group iterative upgrade based on deep learning in this invention.

[0084] Figure 13 This is a logical block diagram of the cloud-based group iterative upgrade module based on deep learning, which is the subject of this invention.

[0085] Figure 14 This is a schematic diagram of the network structure for group local single-machine training and iterative upgrading based on deep learning, as described in this invention.

[0086] Figure 15 This is a schematic diagram of the network structure of the present invention, which is based on deep learning for group local distributed and group cloud iterative upgrade.

[0087] Figure 16 This is a schematic diagram of the structure of a computer system used to implement the embodiments of the methods, systems, and apparatus of the present invention.

[0088] It should be noted that, Figure 14 and Figure 15 For illustration purposes only. The number of devices is not limited to 5 or more. The number of devices for group iterative upgrade can be 2, 3 or more. Detailed Implementation

[0089] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. It should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0090] See Figure 1 The first embodiment of the deep learning-based method for enhancing dark areas of endoscopic images according to the present invention includes the following steps:

[0091] The raw images output from the endoscope are acquired and input into the dark area image enhancement network to obtain the dark area enhanced endoscope images.

[0092] The training method for the dark area image enhancement network is as follows:

[0093] Step S100: Using an endoscope, images of biological tissue are acquired under both low-light and normal-light conditions, and then segmented into... The pixel size is defined as follows: the image acquired by the endoscope in low light is used as the original image, and the image acquired by the endoscope in normal light is used as the target image (label).

[0094] In some preferred embodiments, the original image and the target image (label) are acquired using the following method:

[0095] Step S110: Using an endoscope, images of biological tissue are acquired under low-light conditions and segmented into... Pixel size, used as the original image;

[0096] Step S120: Using an endoscope, images of biological tissue are acquired under normal lighting conditions and segmented into... Pixel size, used as the target image (label);

[0097] Step S200: Using the original endoscope image and the target image (label) as a training set, construct and train a dark area image enhancement network.

[0098] The images obtained by S100 are randomly divided into training, validation, and test sets according to a reasonable ratio (here, 8:1:1) for model training. The training process utilizes a computing platform equipped with an NVIDIA graphics card (Tesla A800 with 80GB). The learning rate of the neural network is set to a reasonable value (here, it can be set to...). The optimizer used is the AdamW optimizer, and the optimizer parameters are set to reasonable values ​​(here, it can be set to...). , ).

[0099] See Figure 2 A dark area image enhancement network is constructed using a layer decomposition network module, a resolution restoration network module, and an illumination adjustment network module.

[0100] Input variables operate convolution kernel Number of output channels Step length Output RGB Convolution + Activation 3×3 32 1 Decomconv1 Decomconv1 Max pooling layer 2×2 32 2 Decompool1 Decompool1 Convolution + Activation 3×3 64 1 Decomconv2 Decomconv2 Max pooling layer 2×2 64 2 Decompool2 Decompool2 Convolution + Activation 3×3 128 1 Decomconv3 Decomconv3 Deconvolution 2×2 64 2 Decomup1 Decomup1,Decomconv2 splicing - 128 - Decomconcat1 Decomconcat1 Convolution + Activation 3×3 64 1 Decomconv4 Decomconv4 Deconvolution 2×2 32 2 Decomup2 Decomup2, Decomconv1 splicing - 64 - Decomconcat2 Decomconcat2 Convolution + Activation 3×3 32 1 Decomconv5 Decomconv5 convolution 3×3 3 1 Decomconv6 Decomconv6 activation - 3 - Decom_Reflectance Decomconv1 Convolution + Activation 3×3 32 1 Decom_i_conv1 Decom_i_conv1,Decomconv5 splicing - 64 - Decom_i_conv2 Decom_i_conv2 convolution 3×3 1 1 Decom_i_conv3 Decom_i_conv3 activation - 1 - Decom Illumination

[0101] Table 1

[0102] Referring to Table 1, the layer decomposition network module contains two branches, corresponding to the resolution branch and the illumination branch, respectively. The resolution branch adopts a five-layer U-shaped network structure, followed by convolutional network layers and a sigmoid activation function layer. The illumination branch consists of two convolutional network layers connected sequentially with an activation function layer, followed by a convolutional layer to receive feature maps from the reflectivity branch, and finally a sigmoid activation function layer.

[0103] Input variables operate convolution kernel Number of output channels Step length Output Decomi_conv3,Decomconv5 splicing - 33 - REconcat1 REconcat1 Convolution + Activation 3×3 32 1 REconv11 REconv11 Convolution + Activation 3×3 32 1 REconv12 REconv12 Max pooling layer 2×2 32 2 REpool1 REpool1 Convolution + Activation 3×3 64 1 REconv21 REconv21 Convolution + Activation 3×3 64 1 REconv22 REconv22 Max pooling layer 2×2 64 2 REpool2 REpool2 Convolution + Activation 3×3 128 1 REconv31 REconv31 Convolution + Activation 3×3 128 1 REconv32 REconv32 Max pooling layer 2×2 128 2 REpool3 REpool3 Convolution + Activation 3×3 256 1 REconv41 REconv41 Convolution + Activation 3×3 256 1 REconv42 REconv42 Max pooling layer 2×2 256 2 REpool4 REpool4 Convolution + Activation 3×3 512 1 REconv51 REconv51 Convolution + Activation 3×3 512 1 REconv52 REconv52 Deconvolution 2×2 256 2 REup1 REup1,REconv42 splicing - 512 - REconcat2 REconcat2 Convolution + Activation 3×3 256 1 REconv61 REconv61 Convolution + Activation 3×3 256 1 REconv62 REconv62 Deconvolution 2×2 128 2 REup2 REup2,REconv32 splicing - 256 - REconcat3 REconcat3 Convolution + Activation 3×3 128 1 REconv71 REconv71 Convolution + Activation 3×3 128 1 REconv72 REconv72 Deconvolution 2×2 64 2 REup3 REup3,REconv22 splicing - 128 - REconcat4 REconcat4 Convolution + Activation 3×3 64 1 REconv81 REconv81 Convolution + Activation 3×3 64 1 REconv82 REconv82 Deconvolution 2×2 32 2 REup4 REup4,REconv12 splicing - 64 - REconcat5 REconcat5 Convolution + Activation 3×3 32 1 REconv91 REconv91 Convolution + Activation 3×3 256 1 REconv92 REconv92 convolution 3×3 3 1 REconv10 REconv10 activation - 3 - RErefletance

[0104] Table 2

[0105] Referring to Table 2, the resolution restoration network has a multi-layer (five or more) U-shaped network structure, followed by convolutional network layers and sigmoid activation function layers.

[0106] Input variables operate convolution kernel Number of output channels Step length Output Decom_illumination,Ratio splicing - 2 - Adjust_concat1 Adjust_concat1 Convolution + Activation 3×3 32 1 Adjust_conv1 Adjust_conv1 Convolution + Activation 3×3 32 1 Adjust_conv2 Adjust_conv2 Convolution + Activation 3×3 32 1 Adjust_conv3 Adjust_conv3 convolution 3×3 1 1 Adjust_conv4 Adjust_conv4 activation - 1 - Adjust_illumination

[0107] Table 3

[0108] Referring to Table 3, the illumination adjustment network is lightweight, consisting of three convolutional layers and one sigmoid activation function layer.

[0109] In some preferred embodiments, the training method for the dark area image enhancement network is as follows:

[0110] Step S210, the original image The images are input into a layer decomposition network to obtain low-light training images. and high-light training images ;

[0111] Step S220, use the low-light training images The image is input into the reflectivity reconstruction network to obtain the first training image. ;

[0112] Step S230, use the high-light training image The image is input into the illumination adjustment network to obtain the second training image. ;

[0113] Step S240: The dark area enhancement image is calculated using the following formula:

[0114]

[0115] In some preferred embodiments, the training method of the dark region enhancement network further includes:

[0116] Step S250: Calculate the mixture loss function of the layer decomposition network. And according to the hybrid loss function The layer decomposition network is modified. The mixture loss function... It can be expressed as

[0117]

[0118] in:

[0119]

[0120]

[0121]

[0122]

[0123] Original image; The reflectance component of the original image; The reflectance component of the original image; and The illumination components are those of the low-light image and the high-light image, respectively; and These are the reflectance components of the low-light and high-light images, respectively.

[0124] Step S260: Calculate the hybrid loss function of the reflectivity reconstruction network. And according to the hybrid loss function The reflector reconstruction network is modified. The hybrid loss function... It can be expressed as

[0125]

[0126] It is an image structure similarity measurement function; and These are the reflectance components of the low-light and high-light images, respectively.

[0127] Step S270: Calculate the hybrid loss function of the illumination adjustment network. And according to the hybrid loss function The illumination adjustment network is modified. The hybrid loss function... It can be expressed as

[0128]

[0129] The reflectance component of the original image; Enhance the reflectance component of the image in dark areas.

[0130] Step S300: Input the original image acquired by the endoscope into the trained dark area image enhancement network, and output the enhanced dark area image.

[0131] In step S400, the original endoscope image is input into the dark area enhancement network self-group iterative upgrade device to perform online correction on the dark area enhancement network.

[0132] See Figure 3 The self-group iterative upgrade device includes a single-machine self-iterative upgrade module and a networked group iterative upgrade module. The difference between self-upgrade and group upgrade lies in whether the training information (dataset, dataset size, dataset quality, etc.) originates from a local single machine (self-upgrade) or a networked group (group upgrade). The group iterative upgrade module includes two functions: local group iterative upgrade and cloud group iterative upgrade. The difference between local group upgrade and cloud group upgrade lies in whether the training process is completed locally or in the cloud. The training device can be local, in the cloud, or both. The local group iterative upgrade includes two functions: local group single-machine training iterative upgrade and local group distributed training iterative upgrade. The local group single-machine training iterative upgrade includes two functions: local group single-machine training with shared dataset iterative upgrade (without sharing training weights) and local group single-machine training with shared dataset and weights iterative upgrade.

[0133] The deep learning-based self-iterative upgrade device includes:

[0134] Storage module for collecting raw endoscopic images input to the dark region enhancement network;

[0135] The expert annotation module is used to annotate the raw endoscopic images in the storage module and generate corresponding labels; the expert annotation module includes a camera with light source supplementation.

[0136] The neural network fine-tuning module is used to correct and fine-tune the dark area enhancement network based on the images collected by the storage module;

[0137] See Figure 4 , Figure 5 The self-iterative upgrade step based on the self-iterative upgrade device is to input the original endoscope image into the dark region enhancement network and train the dark region enhancement network.

[0138] The deep learning-based group iterative upgrade device includes:

[0139] Storage module for collecting raw endoscopic images input to the dark region enhancement network;

[0140] The expert annotation module is used to annotate the raw endoscopic images in the storage module and generate corresponding labels; the expert annotation module includes a camera with light source supplementation.

[0141] The neural network fine-tuning module is used to correct and fine-tune the dark area enhancement network based on the images collected by the storage module;

[0142] The network service module is used for communication and data transmission with the cloud control center.

[0143] See Figure 6 , Figure 7 , Figure 14 The group-based local single-machine training shared dataset iterative upgrade based on the group iterative upgrade module includes:

[0144] The training dataset comes from all networked devices;

[0145] Raw endoscopic images shared by several other networked devices are downloaded via a network module and input into the dark region enhancement network of the current local device. The network is trained locally to obtain corrected dark region enhancement network weights for use by the current local device.

[0146] See Figure 8 , Figure 9 , Figure 14 The group-based local single-machine training shared dataset and weight iterative upgrade based on the group iterative upgrade module includes:

[0147] The training dataset comes from all networked devices;

[0148] Prerequisite: All networked devices must use the same training model software;

[0149] Raw endoscopic images from several other networked devices are downloaded to the current local device via a network module and input into the dark region enhancement network of the current local device. The network is then trained to obtain corrected dark region enhancement network weights, which are then updated and shared in real time with all networked devices for their use.

[0150] See Figure 10 , Figure 11 , Figure 15 The group-based local distributed training iterative upgrade based on the group iterative upgrade module includes:

[0151] Training information (dataset size, dataset quality, etc.) comes from all networked devices;

[0152] Prerequisites: Either all networked devices use the same training model software (which requires knowledge of the dataset size for each device), or the quantitative differences in the parameters of the training datasets generated by each device are known (which requires knowledge of the dataset quality for each device).

[0153] The raw endoscopic images from each local device are input into the local device's dark region enhancement network for training, resulting in corrected dark region enhancement network weights. The corrected dark region enhancement network weights from several networked devices are then uploaded to the cloud control center for global reduction (AllReduce). If all networked devices use the same training model software, global reduction is performed based on the size of each device's dataset. If the networked devices use different training model software and the quantitative differences in the dataset parameters of each device are known, global reduction is performed based on the quantitative differences in the quality of each device's dataset. This yields updated dark region enhancement network weights.

[0154] See Figure 12 , Figure 13 , Figure 15 The group cloud-based iterative upgrade based on the group iterative upgrade module includes:

[0155] Prerequisite: The training model software must be identical across all networked devices and cloud-based devices;

[0156] Raw endoscopic images from several networked devices are transmitted to the cloud control center via a network service module; the cloud control center aggregates the received raw endoscopic images and trains the dark region enhancement network; the local network service module downloads and synchronously updates the dark region enhancement network weights for the current device.

[0157] A second embodiment of the present invention provides a method for enhancing dark areas of endoscopic images based on deep learning. The system is a method for enhancing dark areas of endoscopic images based on deep learning, and includes: an acquisition module and a training module.

[0158] The acquisition module is configured to acquire the original endoscope image, acquire the biological tissue image obtained by the endoscope under very low light intensity as the original image, and acquire the biological tissue image again by the endoscope under normal light intensity as the target image (label).

[0159] The training method for the dark area image enhancement network is as follows:

[0160] The training module is configured to acquire dark area enhanced images;

[0161] The acquired original images and corresponding target images (labels) are used as a dataset to construct and train a dark area image enhancement network.

[0162] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the storage device and processing device described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0163] It should be noted that the endoscopic imaging system of the deep learning-based endoscopic image dark area enhancement method provided in the above embodiments is only an example of the above functional module division. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0164] An electronic device according to a third embodiment of the present invention includes: at least one processor and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to implement the above-described method for enhancing dark areas of endoscopic images based on deep learning.

[0165] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions for execution by the computer to implement the above-described method for enhancing dark areas of endoscopic images based on deep learning.

[0166] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the storage device and processing device described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0167] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.

[0168] The following is for reference. Figure 16 It shows a schematic diagram of the structure of a computer system for implementing embodiments of the methods, systems, and apparatus of the present invention. Figure 16 The server shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0169] like Figure 16 As shown, the computer system includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in Read Only Memory (ROM) 502 or programs loaded from storage section 508 into Random Access Memory (RAM) 503. RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0170] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0171] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined in the methods of this invention. It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0172] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0173] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0174] The terms “first”, “second”, etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.

[0175] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.

[0176] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for enhancing dark areas in endoscopic images based on deep learning, characterized in that, The method includes: The raw images output from the endoscope are acquired and input into the dark area image enhancement network to obtain the dark area enhanced endoscope image; The training method for the dark area image enhancement network is as follows: Step S100: Using an endoscope, images of biological tissue are acquired under both low-light and normal-light conditions, and then segmented into... The pixel size is defined as follows: the image acquired by the endoscope in low light is used as the original image, and the image acquired by the endoscope in normal light is used as the target image (label). Step S200: Using the original endoscope image and the target image (label) as a training set, construct and train a dark area image enhancement network; Step S300: Input the original image obtained by the endoscope into the trained dark area image enhancement network, and output the enhanced dark area image.

2. The method for enhancing dark areas of endoscopic images based on deep learning according to claim 1, characterized in that, The original image and the target image (label) were acquired using the following method: Step S110: Using an endoscope, images of biological tissue are acquired under low-light conditions and segmented into... Pixel size, used as the original image; Step S120: Using an endoscope, images of biological tissue are acquired under normal lighting conditions and segmented into... The pixel size is used as the target image (label).

3. The method for enhancing dark areas of endoscopic images based on deep learning according to claim 1, characterized in that, The low-light image enhancement network is constructed as follows: Referring to Figure 2, a dark area image enhancement network is constructed using a layer decomposition network module, a resolution restoration network module, and an illumination adjustment network module. As shown in Table 1, the layer decomposition network module contains two branch networks, corresponding to resolution and illumination respectively. The resolution branch adopts a five-layer U-shaped network structure, followed by a convolutional network layer and a sigmoid activation function layer. The illumination branch consists of two convolutional network layers and an activation function layer connected sequentially, followed by a convolutional layer to receive feature maps from the reflectivity branch, and finally a sigmoid activation function layer. Referring to Table 2, the resolution restoration network has a multi-layer (five or more) U-shaped network structure, followed by convolutional network layers and sigmoid activation function layers; Referring to Table 3, the illumination adjustment network is lightweight, consisting of three convolutional layers and one sigmoid activation function layer.

4. The method for enhancing dark areas of endoscopic images based on deep learning according to claim 1, characterized in that, The training method for the dark area image enhancement network is as follows: Step S210, the original image The images are input into a layer decomposition network to obtain low-light training images. and high-light training images ; Step S220, use the low-light training images The image is input into the reflectivity reconstruction network to obtain the first training image. ; Step S230, use the high-light training image The image is input into the illumination adjustment network to obtain the second training image. ; Step S240: The dark area enhancement image is calculated using the following formula: 。 5. The dark area image enhancement network according to claim 4, characterized in that, The blending loss function of the layer decomposition network This can be expressed as: in: Original image; The reflectance component of the original image; The reflectance component of the original image; and The illumination components are those of the low-light image and the high-light image, respectively; and These are the reflectance components of the low-light and high-light images, respectively.

6. The dark area image enhancement network according to claim 4, characterized in that, The hybrid loss function of the reflectivity reconstruction network This can be expressed as: It is an image structure similarity measurement function; and These are the reflectance components of the low-light and high-light images, respectively.

7. The dark area image enhancement network according to claim 4, characterized in that, The hybrid loss function of the illumination adjustment network This can be expressed as: The reflectance component of the original image; Enhance the reflectivity component of the image in dark areas.

8. A self-group iterative upgrade device based on deep learning, characterized in that, The self-group iterative upgrade device includes a single-machine self-iterative upgrade module and a networked group iterative upgrade module. The difference between self-upgrade and group upgrade lies in whether the training information (dataset, dataset size, dataset quality, etc.) originates from a local single machine (self-upgrade) or a networked group (group upgrade). The group iterative upgrade module includes two functions: local group iterative upgrade and cloud group iterative upgrade. The difference between local group upgrade and cloud group upgrade lies in whether the training process is completed locally or in the cloud. The training device can be local, in the cloud, or both. The local group iterative upgrade includes two functions: local group single-machine training iterative upgrade and local group distributed training iterative upgrade. The local group single-machine training iterative upgrade includes two functions: local group single-machine training with shared dataset iterative upgrade (without sharing training weights) and local group single-machine training with shared dataset and weights iterative upgrade.

9. The self-iterative upgrade module based on deep learning according to claim 8, characterized in that, The deep learning-based self-iterative upgrade module includes: Storage module for collecting raw endoscopic images input to the dark region enhancement network; The expert annotation module is used to annotate the raw endoscopic images in the storage module and generate corresponding labels; the expert annotation module includes a camera with light source supplementation; The neural network fine-tuning module is used to correct and fine-tune the dark area enhancement network based on the images collected by the storage module; The self-iterative upgrade step based on the self-iterative upgrade device is to input the original endoscope image into the dark region enhancement network and train the dark region enhancement network.

10. The deep learning-based population iterative upgrade module according to claim 8, characterized in that, The deep learning-based group iterative upgrade module includes: Storage module for collecting raw endoscopic images input to the dark region enhancement network; The expert annotation module is used to annotate the raw endoscopic images in the storage module and generate corresponding labels; the expert annotation module includes a camera with light source supplementation; The neural network fine-tuning module is used to correct and fine-tune the dark area enhancement network based on the images collected by the storage module; The network service module is used for communication and data transmission with other devices or cloud control centers.

11. The group local single-machine training shared dataset iterative upgrade function of the deep learning-based group iterative upgrade module according to claim 8, characterized in that, The iterative upgrade of the shared local single-machine training dataset based on the group iterative upgrade module includes: The training dataset comes from all networked devices; Raw endoscopic images shared by several other networked devices are downloaded via a network module and input into the dark region enhancement network of the current local device. The network is trained locally to obtain corrected dark region enhancement network weights for use by the current local device.

12. The group local single-machine training shared dataset and weight iterative upgrade function of the deep learning-based group iterative upgrade module according to claim 8, characterized in that, The group-based local single-machine training shared dataset and weight iterative upgrade based on the group iterative upgrade module includes: The training dataset comes from all networked devices; Prerequisite: All networked devices must use the same training model software; Raw endoscopic images from several other networked devices are downloaded to the current local device via a network module and input into the dark region enhancement network of the current local device. The network is then trained to obtain corrected dark region enhancement network weights, which are then updated and shared in real time with all networked devices for their use.

13. The group-local distributed training iterative upgrade function of the deep learning-based group iterative upgrade module according to claim 8, characterized in that, The group-based local distributed training iterative upgrade based on the group iterative upgrade module includes: Training information (dataset size, dataset quality, etc.) comes from all networked devices; Prerequisites: Either all networked devices use the same training model software (which requires knowledge of the dataset size for each device), or the quantitative differences in the parameters of the training datasets generated by each device are known (which requires knowledge of the dataset quality for each device). The raw endoscopic images from each local device are input into the local device's dark region enhancement network for training, resulting in corrected dark region enhancement network weights. The corrected dark region enhancement network weights from several networked devices are then uploaded to the cloud control center for global reduction (AllReduce). If all networked devices use the same training model software, global reduction is performed based on the size of each device's dataset. If the networked devices use different training model software and the quantitative differences in the dataset parameters of each device are known, global reduction is performed based on the quantitative differences in the quality of each device's dataset. This yields updated dark region enhancement network weights.

14. The group cloud-based iterative upgrade function of the group iterative upgrade module based on deep learning according to claim 8, characterized in that, The group cloud-based iterative upgrade based on the group iterative upgrade module includes: Prerequisite: The training model software must be identical across all networked devices and cloud-based devices; Raw endoscopic images from several networked devices are transmitted to the cloud control center via a network service module; the cloud control center aggregates the received raw endoscopic images and trains the dark region enhancement network; the local network service module downloads and synchronously updates the dark region enhancement network weights for the current device.

Images