Image enhancement method and apparatus, electronic device, and storage medium

Abstract

Provided in the present application are an image enhancement method and apparatus, an electronic device, and a storage medium, relating to the technical field of image processing. The method comprises: acquiring a medical data set, and collecting speckle images by means of an optical fiber endoscope; generating a synthetic data set on the basis of the speckle images and medical images in the medical data set; and performing adversarial training on a deep learning model on the basis of the synthetic data set and the medical data set to generate an image reconstruction model, so as to perform image reconstruction on endoscopic images on the basis of the image reconstruction model to obtain image reconstruction results. The present application solves the problem of poor image enhancement effect of image enhancement in the related art.

Description

Image enhancement methods, apparatus, electronic devices and storage media Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image enhancement method, apparatus, electronic device, and storage medium. Background Technology

[0002] Endoscopes are widely used in gastrointestinal (GI) diagnosis and minimally invasive surgery. They enable minimally invasive examinations and allow for deep penetration into the body to examine internal tissues, making them highly valuable. In the field of flexible endoscopes, fiber optic bundles are used for image transmission. Each fiber in the bundle transmits one image point, and the end face of the entire bundle forms the endoscopic image to be transmitted. Due to the inherent structural characteristics of fiber optic bundles, transmitting images using fiber optic bundles introduces a honeycomb-like speckle pattern into the image. Furthermore, in practical applications, the fiber pattern may contain various noises and distortions. Therefore, image enhancement is necessary to improve the image quality of the endoscopic images.

[0003] Existing image enhancement methods, including traditional imaging techniques, require extensive manual adjustments to optical experimental equipment to obtain a clear image. This process is extremely complex and involves numerous formula derivations, making it time-consuming and labor-intensive.

[0004] In addition, computational imaging methods include deep learning methods, which use a single fiber image as input and directly train a neural network to map the fiber image and CCD image to remove fiber speckle, or use interpolation to remove cellular patterns. Then, the neural network is applied to learn and recover the mapping from the fiber image to the ground truth (GT) image to further improve resolution. However, these methods are usually based on the Minist dataset of handwritten digit images or the ImageNet dataset of natural scenes. The datasets have limited scene representation, and deep learning methods often result in poor model generalization, performing well only on very small, specific datasets. They are not suitable for actual imaging images of millimeter-scale fibers, so the results are poor in actual tests, with limited resolution improvement, and they cannot effectively remove speckle. Furthermore, they are only applicable to specific scenarios, and the image processing effect is greatly affected by uneven lighting and fiber physical parameters, making it difficult to simultaneously achieve edge feature noise removal and resolution improvement.

[0005] As can be seen from the above, the problem of how to improve the image enhancement effect still needs to be solved.

[0006] Summary of the Invention

[0007] This application provides an image enhancement method, apparatus, electronic device, and storage medium, which can solve the problem of poor image enhancement effect in related technologies. The technical solutions are as follows:

[0008] According to one aspect of this application, an image enhancement method is characterized by comprising:

[0009] Acquire medical datasets and capture speckle images using a fiber optic endoscope;

[0010] A synthetic dataset is generated based on the speckle image and the medical images in the medical dataset, wherein the synthetic dataset includes synthetic images corresponding to each medical image in the medical dataset;

[0011] The deep learning model is adversarially trained based on the synthetic dataset and the medical dataset to generate an image reconstruction model, which is then used to reconstruct endoscopic images to obtain image reconstruction results.

[0012] According to one aspect of this application, an image enhancement apparatus is characterized by comprising:

[0013] The image acquisition module is used to acquire medical datasets and collect speckle images via a fiber optic endoscope;

[0014] A dataset generation module is used to generate a synthetic dataset based on the speckle image and the medical images in the medical dataset, wherein the synthetic dataset includes synthetic images corresponding to each medical image in the medical dataset;

[0015] The training module is used to perform adversarial training on the deep learning model based on the synthetic dataset and the medical dataset, generate an image reconstruction model, and perform image reconstruction on the endoscopic image based on the image reconstruction model to obtain the image reconstruction result.

[0016] In one exemplary embodiment, the image acquisition module includes:

[0017] The background image acquisition unit is used to acquire a pure white background image;

[0018] The image acquisition unit is used to acquire images using a fiber optic endoscope with the pure white background image as the target, and obtain speckle images.

[0019] In one exemplary embodiment, the dataset generation module includes:

[0020] A preprocessing unit is used to preprocess the speckle image and the medical images in the medical dataset;

[0021] The fusion unit is used to multiply and fuse the speckle image with the medical image to generate a synthetic image, thereby obtaining a synthetic dataset.

[0022] In one exemplary embodiment, the deep learning model includes a generator network and a discriminator network, wherein the generator network is a convolutional neural network structure including an encoder and a decoder connected by multi-scale skip connections, and the discriminator network is a convolutional neural network structure capable of performing similarity discrimination on at least two input images and generating a discrimination result.

[0023] In one exemplary embodiment, the training module includes:

[0024] An image reconstruction unit is used to input the synthetic image from the synthetic dataset into the generator network, and perform image reconstruction on the synthetic image based on the encoder and decoder in the generator network to obtain a reconstructed image.

[0025] The discrimination unit is used to input the reconstructed image and the medical images in the medical dataset into the discrimination network for similarity discrimination and to obtain the discrimination result;

[0026] The update unit is used to update the parameters of the generator network and the discriminator network based on the discrimination result, complete the training of the generator network and the discriminator network, and obtain the trained image reconstruction model.

[0027] In one exemplary embodiment, the image reconstruction unit includes:

[0028] The feature extraction subunit is used to perform multi-level feature extraction on the synthesized image based on the encoder in the generator network to obtain image features;

[0029] The feature fusion subunit is used to perform feature fusion on the image features based on the decoder in the generator network to obtain a reconstructed image.

[0030] In one exemplary embodiment, the updating unit includes:

[0031] An image determination subunit is used to determine the medical image corresponding to the synthetic image based on the synthetic image source of the reconstructed image;

[0032] The loss value calculation subunit is used to input the reconstructed image and the medical image into a loss function and calculate the loss value corresponding to the reconstructed image.

[0033] The training subunit is used to determine whether the discrimination result is correct based on the correspondence between the reconstructed image and the medical image; if the discrimination result correctly identifies the image source of the input discrimination network, the network parameters corresponding to the discrimination network are updated based on the loss value; if the discrimination result incorrectly identifies the image source of the input discrimination network, the network parameters corresponding to the generator network are updated based on the loss value; training stops when the loss function converges, and the image reconstruction model is obtained.

[0034] According to one aspect of this application, an electronic device includes at least one processor and at least one memory, wherein computer-readable instructions are stored on the memory; the computer-readable instructions are loaded and executed by the processor, causing the electronic device to implement the image enhancement method as described above.

[0035] According to one aspect of this application, a storage medium has computer-readable instructions stored thereon, which are loaded and executed by a processor to implement the image enhancement method as described above.

[0036] According to one aspect of this application, a computer program product includes computer-readable instructions stored in a storage medium, wherein a processor of an electronic device reads the computer-readable instructions from the storage medium, loads and executes the computer-readable instructions, causing the electronic device to implement the image enhancement method as described above.

[0037] The beneficial effects of the technical solution provided in this application are:

[0038] In the above technical solution, a synthetic dataset is generated by fusing speckle images and medical images in a medical dataset. The synthetic dataset is then used to train an image reconstruction model that can reconstruct images from endoscopic images. This allows the generated image reconstruction model to take into account both the removal of speckle in endoscopic images and the white balance of image color during the image reconstruction process, thereby improving the resolution and effectiveness of image reconstruction. Furthermore, the model is more applicable to real-world scenarios and improves its generalization ability, thus effectively solving the problem of poor image enhancement effects in related technologies. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0040] Figure 1 is a schematic diagram of the implementation environment according to this application;

[0041] Figure 2 is a flowchart illustrating an image enhancement method according to an exemplary embodiment;

[0042] Figure 3 is a flowchart of step 210 in one embodiment of the embodiment corresponding to Figure 2;

[0043] Figure 4 is a flowchart of step 230 in one embodiment of the embodiment corresponding to Figure 2;

[0044] Figure 5a is a flowchart of step 250 in one embodiment of the embodiment corresponding to Figure 2;

[0045] Figure 5b is a schematic diagram of a generative network structure according to an exemplary embodiment;

[0046] Figure 5c is a schematic diagram of a discriminant network structure according to an exemplary embodiment;

[0047] Figure 6 is a flowchart of step 251 in one embodiment of the embodiment corresponding to Figure 5a;

[0048] Figure 7 is a schematic diagram of the encoder and decoder structure according to an exemplary embodiment;

[0049] Figure 8 is a flowchart of step 255 in one embodiment of the embodiment corresponding to Figure 5a;

[0050] Figure 9 is a schematic diagram of the specific implementation of an image enhancement method in an application scenario;

[0051] Figure 10 is a structural block diagram of an image enhancement device according to an exemplary embodiment;

[0052] Figure 11 is a hardware structure diagram of a server according to an exemplary embodiment;

[0053] Figure 12 is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0054] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0055] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0056] As mentioned earlier, existing technologies are typically based on the Minist dataset of handwritten digit images or the ImageNet dataset of natural scenes. These datasets have limited scene representation, and the deep learning methods result in poor model generalization. They often perform well only on very small, specific datasets and are not suitable for actual imaging images of millimeter-scale optical fibers. Therefore, the actual test results are poor, the resolution improvement is limited, and speckle cannot be effectively removed. Furthermore, they are only applicable to certain specific scenes, and the image processing effect is greatly affected by uneven lighting and the physical parameters of the optical fiber. As a result, the model has poor generalization and it is difficult to simultaneously achieve edge feature noise removal and resolution improvement, leading to a decline in image enhancement effect.

[0057] As can be seen from the above, the related technologies still suffer from poor image enhancement effects.

[0058] Therefore, the image enhancement method provided in this application can effectively improve the image enhancement effect. Accordingly, the image enhancement method is applicable to an image enhancement device, which can be deployed in an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, such as a desktop computer, a laptop computer, a server, etc.

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0060] Figure 1 is a schematic diagram of an implementation environment involved in an image feature extraction method. It should be noted that this implementation environment is merely an example adapted to the present invention and should not be considered as providing any limitation on the scope of the invention.

[0061] The implementation environment includes a data acquisition terminal 110 and a server terminal 130.

[0062] Specifically, the acquisition end 110 can also be considered as an image acquisition device, such as a fiber optic endoscope.

[0063] Server 130 can be an electronic device such as a desktop computer, laptop computer, or server, or it can be a computer cluster consisting of multiple servers, or even a cloud computing center consisting of multiple servers. Server 130 is used to provide backend services, such as, but not limited to, image enhancement services.

[0064] The server 130 and the acquisition terminal 110 establish a network communication connection in advance via wired or wireless means, and data transmission between the server 130 and the acquisition terminal 110 is realized through this network communication connection. The transmitted data includes, but is not limited to, endoscopic images, etc.

[0065] In one application scenario, through the interaction between the acquisition terminal 110 and the server terminal 130, the acquisition terminal 110 acquires an endoscope image and uploads the endoscope image to the server terminal 130 to request the server terminal 130 to provide image enhancement services.

[0066] For server 130, after receiving the image to be processed uploaded by acquisition terminal 110, it calls the image enhancement service to enhance the endoscopic image, obtain the image reconstruction result, and improve the image enhancement effect, thereby solving the problem of poor image enhancement effect in related technologies.

[0067] Please refer to Figure 2. This application embodiment provides an image enhancement method. The method is applicable to electronic devices, which can be the server 130 in the implementation environment shown in Figure 1, or a desktop computer, laptop computer, server, etc.

[0068] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0069] As shown in Figure 2, the method may include the following steps:

[0070] Step 210: Obtain the medical dataset and acquire speckle images using a fiber optic endoscope.

[0071] The medical dataset includes high-resolution medical images from the medical field, and speckle images are endoscopic images with honeycomb speckle patterns actually acquired through a fiber optic endoscope.

[0072] In one possible implementation, the medical dataset is the hyper-kVasir dataset.

[0073] Step 230: Generate a synthetic dataset based on speckle images and medical images from the medical dataset.

[0074] The synthetic dataset includes synthetic images, which are generated by fusing honeycomb speckle images from speckle images with medical images from medical datasets to produce low-resolution images with fiber optic honeycomb speckle that combine the characteristics of both.

[0075] Step 250: The deep learning model is adversarially trained based on the synthetic dataset and the medical dataset to generate an image reconstruction model, which is then used to reconstruct the endoscopic images to obtain the image reconstruction results.

[0076] It can be understood that endoscopic images are target images acquired during the practical application of endoscopy that require image reconstruction, such as human images acquired during gastrointestinal (GI) diagnosis and minimally invasive surgery. The image reconstruction result is a high-quality image generated by the trained image reconstruction model after reconstructing the endoscopic image.

[0077] It should be noted that, in an exemplary embodiment, the deep learning model includes a generator network and a discriminator network, wherein the generator network is a convolutional neural network structure including an encoder and a decoder connected by multi-scale jump connections, and the discriminator network is a convolutional neural network structure capable of performing similarity discrimination on at least two input images and generating discrimination results.

[0078] Specifically, the synthetic image corresponding to the medical image is input into a deep learning network. The generative network eliminates the honeycomb speckle in the synthetic image and performs image reconstruction to obtain a high-quality reconstructed image. The discriminative network judges the difference between the reconstructed image and the original high-resolution medical image, thereby realizing the training of the adversarial network. Through continuous iteration, the performance of the generative and discriminative networks is continuously improved, and finally, an image reconstruction model capable of high-quality image reconstruction of the synthetic network is output.

[0079] At this point, the endoscope image generated in the actual application of the endoscope is acquired and input into the image reconstruction model. The image reconstruction result is generated by the generative network in the image reconstruction model to reconstruct the endoscope image.

[0080] In one possible implementation, images from the medical dataset and the synthetic dataset are divided into training and test sets in an 8:2 ratio. The deep learning model is adversarially trained using high-resolution medical images and low-resolution synthetic images containing honeycomb patterns in the training set to obtain an image reconstruction model. Then, the low-resolution synthetic images in the test set are input into the deep learning model to obtain the image reconstruction results. The training effect of the image reconstruction model is judged based on the image reconstruction effect of the image reconstruction results.

[0081] Through the above process, a synthetic dataset is generated by fusing speckle images and medical images from a medical dataset. The synthetic dataset is then used to train an image reconstruction model that can reconstruct endoscopic images. This allows the generated image reconstruction model to better adapt to endoscopic images in different scenarios in real-world environments. The model exhibits good generalization, high robustness, and good image enhancement effects.

[0082] In an exemplary embodiment, as shown in FIG3, step 210 may include the following steps:

[0083] Step 211: Obtain a pure white background image.

[0084] Among them, a pure white background image is a white background image without any image interference.

[0085] Step 213: Image acquisition is performed using a fiber optic endoscope with a pure white background as the target to obtain a speckle image.

[0086] It is understandable that by acquiring images from a pure white background using a fiber optic endoscope, the acquired speckle image is not affected by the color of the background image and can fully reflect the inherent interference such as honeycomb speckle and noise during the image acquisition process of the fiber optic endoscope.

[0087] In one possible implementation, a fiber optic endoscope image acquisition test bench is built, the endoscope is connected to an industrial camera, and then the industrial camera is connected to a computer via a USB cable. This allows the industrial camera to capture images of a pure white background on the test bench through the endoscope and store the obtained speckle images to the computer.

[0088] Through the above process, a speckle image that accurately reflects the characteristics of endoscopic images is obtained, ensuring the accuracy of subsequent image training and improving the image reconstruction effect of the image reconstruction model.

[0089] In one exemplary embodiment, as shown in FIG4, step 230 may include the following steps:

[0090] Step 231: Preprocess the speckle image and the medical images in the medical dataset.

[0091] The preprocessing step involves standardizing and unifying the speckle images and medical images to enable their fusion. For example, if the speckle image size is 1024*1024, the medical images in the medical dataset are standardized to a size of 1024*1024.

[0092] Step 233: Multiply and fuse the speckle image with the medical image to generate a synthetic image and obtain a synthetic dataset.

[0093] By performing pixel-level multiplication on speckle images and medical images, interference features such as honeycomb speckle and noise in the speckle images are extracted and fused into the medical images, thus transforming the high-resolution medical images into low-resolution images with fiber optic honeycomb speckle.

[0094] In one possible implementation, the medical dataset is the hyper-kvasir dataset, and a synthetic dataset hyper-kvasir-fbimage is generated by fusing speckle images and the hyper-kvasir dataset.

[0095] Through the above process, preprocessing ensures the synthesis effect and efficiency of speckle images and medical images. By fusing speckle images and medical images in the medical dataset to generate a synthetic dataset, the training accuracy during subsequent model training is guaranteed, thereby improving the image reconstruction effect and generalization of the image reconstruction model.

[0096] In an exemplary embodiment, as shown in FIG5a, step 250 may include the following steps:

[0097] Step 251: Input the synthetic image from the synthetic dataset into the generator network, and reconstruct the synthetic image based on the encoder and decoder in the generator network to obtain the reconstructed image.

[0098] The generator network accepts low-resolution synthetic images and extracts features from the synthetic images through progressive encoding and decoding, ultimately reconstructing them into high-resolution reconstructed images.

[0099] In one possible implementation, as shown in Figure 5b, the generator network is an FBNet network with a UET structure. The generator network includes an encoder connected to the input, a decoder connected to the encoder via multi-scale hops, and an output connected to the decoder. The initial FBNet network is trained, and the trained parameters are used as the initial parameters of the generator network. During training, the loss function adopts pixel-level MSE loss, and the loss function format is as follows:

[0100] Among them, I HR For medical images, I SR To reconstruct the image, This is the loss value.

[0101] In an exemplary embodiment, as shown in FIG6, step 251 may include the following steps:

[0102] Step 2511: Perform multi-level feature extraction on the synthesized image based on the encoder in the generative network to obtain image features.

[0103] Step 2513: Based on the decoder in the generator network, feature fusion is performed on the image features to obtain the reconstructed image.

[0104] Specifically, as shown in Figure 7, the encoder includes multiple downsampling modules connected by max-pooling layers. Each downsampling module comprises cyclically arranged convolutional layers and activation function layers. These modules progressively downsample the synthesized image, extracting shallow and generated features sequentially. The decoder includes multiple upsampling modules, which consist of transposed convolutional layers, planning function layers, and deep connected layers. The decoder upsamples the shallow and generated features generated by the encoder layer by layer, gradually fusing them to form the reconstructed image. Simultaneously, it connects the feature maps of each layer in the encoder with the corresponding feature maps in the decoder, allowing for the exchange of detailed information between the layers in the decoder and encoder, thus improving the image reconstruction effect.

[0105] In one possible implementation, the activation function in the generator network is the ReLU function.

[0106] Through the above process, image reconstruction based on local and global information in the image is achieved by using a generative network. By using skip connections between the encoder and decoder, the generator can capture rich contextual information at different levels and pass detailed information to the decoder, thereby improving the decoder's accuracy in recovering image details. At the same time, the model has relatively few parameters and a simple structure, which makes training and inference faster and improves image reconstruction efficiency.

[0107] Step 253: Input the reconstructed image and the medical images in the medical dataset into the discriminant network for similarity discrimination to obtain the discrimination result.

[0108] Specifically, as shown in Figure 5c, the discriminant network is a binary classifier with a convolutional neural structure. The discriminant network includes multiple NO modules, each containing a convolutional layer, a batch normalization layer, and an activation function layer. The multiple NO modules receive the reconstructed image generated by the generator network and the corresponding medical image. The difference between the two images is used to determine whether they are real or similar, and finally, a discrimination result is generated.

[0109] In one possible implementation, the activation functions in the discriminant network include the Leakly ReLU function and the Sigmoid function.

[0110] Step 255: Update the parameters of the generator network and the discriminator network based on the discrimination results, complete the training of the generator network and the discriminator network, and obtain the trained image reconstruction model.

[0111] In an exemplary embodiment, as shown in FIG8, step 255 may include the following steps:

[0112] Step 2551: Determine the medical image corresponding to the synthetic image by identifying the source of the synthetic image from the reconstructed image.

[0113] It should be noted that each synthetic image has a corresponding medical image. After image reconstruction of the synthetic image, the generated reconstructed image and the corresponding medical image of the synthetic image form an image pair. The effect of the reconstructed image is evaluated by the difference between the image pairs.

[0114] Step 2553: Input the reconstructed image and the medical image into the loss function to calculate the loss value of the corresponding reconstructed image.

[0115] The loss function used during training is employed to quantify and evaluate the differences between image pairs. Specifically, the perceptual loss function of the generative network includes content loss and adversarial loss, and the calculation formula for the loss function is as follows:

[0116] Wherein, content loss is the content loss, adversarial loss is the adversarial loss, and I HR For medical images, I SR To reconstruct the image.

[0117] The content loss includes VGG loss at the feature map level and color loss in the color space. The VGG loss uses the feature vectors from a pre-trained VGG-19 network.

[0118] The result of the network is the feature map and I generated after passing through a certain layer of the VGG network. HR The loss is calculated using the feature map labels generated by the VGG network.

[0119] The formula for calculating the adversarial loss function during training is as follows:

[0120] It should be noted that the inventors recognized that the Euclidean distance in the RGB space of endoscopic images cannot effectively reflect color distortion. Therefore, during image reconstruction, color loss is introduced into the content loss, and the color distribution of the reconstructed image is constrained by the discrete cosine distance. The formula for calculating the color distribution constraint is as follows:

[0121] In the above process, a perceptual loss function including content loss and adversarial loss is adopted, and the color distribution of the reconstructed image is constrained to make the color distribution of the reconstructed image closer to the color distribution of the original high-resolution image, so that the loss value can better reflect the perceptual similarity between images. This allows the generator network to produce higher discriminative values ​​and improves the model's interconnection effect. When reconstructing images, while removing honeycomb speckle, image white balance can be achieved, realizing the compatibility of thin fiber optic endoscopy with high-quality image acquisition and improving the endoscopic image enhancement effect.

[0122] Step 2555: Determine whether the discrimination result is correct based on the correspondence between the reconstructed image and the medical image; if the discrimination result correctly identifies the image source of the input discrimination network, update the network parameters corresponding to the discrimination network based on the loss value; if the discrimination result incorrectly identifies the image source of the input discrimination network, update the network parameters corresponding to the generation network based on the loss value; until the loss function converges, stop training and obtain the image reconstruction model.

[0123] During training, in order to make the images generated by the generator network as close as possible to the original medical images, and at the same time enable the discriminator network to accurately distinguish between the reconstructed images and the medical images, the generator network and the discriminator network need to be subjected to adversarial attacks. By adversarial attacks on the generator network and the discriminator network, the image reconstruction ability of the generator network and the judgment ability of the discriminator network are improved. By continuously updating the model parameters of the generator network and the discriminator network, the generator network is trained to be closer to the medical images, and the discriminator network is trained to be more discriminative.

[0124] Under the above embodiments, the image reconstruction model is trained based on the fourth image generated by the generative network, thereby improving the image reconstruction capability of the image reconstruction model and stabilizing the training process.

[0125] Figure 9 is a schematic diagram illustrating the specific implementation of an image enhancement method in an application scenario. In this scenario, the speckle image is an image of the fiber bundle acquired by a fiber optic endoscope, and the medical dataset is the hyper-kvasir dataset.

[0126] Medical images from the hyper-kvasir dataset were preprocessed.

[0127] Then, by acquiring fiber bundle images from fiber optic endoscopes, the honeycomb speckle pattern is extracted and multiplied and fused with medical images to generate a synthetic dataset.

[0128] Synthetic images from the synthetic dataset are used to train the deep learning model FBUGAN adversarially, while medical images from the hyper-kVasir dataset are input into the discriminant network for judgment.

[0129] Based on the judgment results of the discriminant network, backpropagation is performed to optimize the generator network and the discriminant network, and a trained image reconstruction model is obtained.

[0130] The endoscopic image is reconstructed using a generative network in the image reconstruction model to obtain the image reconstruction result.

[0131] In this application scenario, the generated image reconstruction model can simultaneously address the removal of speckle in endoscopic images and improve image color white balance during the image reconstruction process, thereby enhancing the resolution of image reconstruction. Furthermore, the applicable scenarios are more realistic, improving the model's generalization ability. This effectively solves the problem of poor image enhancement effects in related technologies.

[0132] The following are embodiments of the apparatus described in this application, which can be used to execute the image enhancement method involved in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of the image enhancement method involved in this application.

[0133] Please refer to Figure 10. In this embodiment of the application, an image enhancement device 1000 is provided, including but not limited to: an image acquisition module 1010, a dataset generation module 1030, and a training module 1050.

[0134] The image acquisition module 1010 is used to acquire medical datasets and collect speckle images through a fiber optic endoscope.

[0135] Dataset generation module 1030 is used to generate synthetic datasets based on speckle images and medical images from medical datasets;

[0136] Training module 1050 is used to perform adversarial training on deep learning models based on synthetic and medical datasets to generate image reconstruction models, which are then used to reconstruct endoscopic images and obtain image reconstruction results.

[0137] It should be noted that the image enhancement device provided in the above embodiments is only illustrated by the division of the above functional modules when performing image enhancement. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the image enhancement device will be divided into different functional modules to complete all or part of the functions described above.

[0138] Furthermore, the image enhancement apparatus and image enhancement method embodiments provided in the above embodiments belong to the same concept, and the specific way in which each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0139] Figure 11 is a schematic diagram of a server structure according to an exemplary embodiment. This server is suitable for server 130 in the implementation environment shown in Figure 1.

[0140] It should be noted that this server is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Nor should this server be interpreted as requiring or needing to have one or more of the components of the exemplary server 2000 shown in Figure 11.

[0141] The hardware structure of the server 2000 can vary greatly depending on the configuration or performance. As shown in Figure 11, the server 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0142] Specifically, power supply 210 is used to provide operating voltage for the various hardware devices on server 2000.

[0143] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. For example, it enables interaction between the acquisition terminal 110 and the server terminal 130 in the implementation environment shown in Figure 1.

[0144] Of course, in other examples adapted in this application, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, as shown in Figure 11, but this is not intended to be a specific limitation.

[0145] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0146] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the server 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0147] Application 253 is a computer-readable instruction based on operating system 251 that performs at least one specific task. It may include at least one module (not shown in FIG11), and each module may contain computer-readable instructions for server 2000. For example, an image enhancement device may be considered as application 253 deployed on server 2000.

[0148] Data 255 can be photos, pictures, etc. stored on a disk, or it can be training image sets, images to be processed, etc., stored in memory 250.

[0149] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby performing operations and processing on massive amounts of data 255 stored in the memory 250. For example, an image enhancement method may be implemented by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0150] Furthermore, this application can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of this application is not limited to any specific hardware circuit, software, or combination thereof.

[0151] Please refer to Figure 12. This application embodiment provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc.

[0152] In Figure 12, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0153] Data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in Figure 12, but this does not indicate that there is only one bus or one type of bus.

[0154] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0155] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0156] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or it may be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired computer-readable instructions in the form of instructions or data structures and accessible by the electronic device 400, but not limited thereto.

[0157] The memory 4003 stores computer-readable instructions, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002.

[0158] When the computer-readable instructions are executed by the processor 4001, they implement the image enhancement methods described in the above embodiments.

[0159] Furthermore, this application provides a storage medium storing computer-readable instructions, which are loaded and executed by a processor to implement the image enhancement method described above.

[0160] This application provides a computer program product including computer-readable instructions stored in a storage medium. The processor of an electronic device reads the computer-readable instructions from the storage medium, loads and executes the computer-readable instructions, thereby enabling the electronic device to implement the image enhancement method described above.

[0161] Compared with related technologies, this application generates a synthetic dataset by fusing speckle images and medical images from a medical dataset. This synthetic dataset is then used to train an image reconstruction model capable of reconstructing endoscopic images. This allows the generated model to better adapt to reconstructing endoscopic images in different scenarios in real-world environments, exhibiting good generalization, robustness, and image enhancement. It obtains speckle images that accurately reflect the characteristics of endoscopic images, ensuring the accuracy of subsequent image training and improving the image reconstruction effect. Preprocessing ensures the synthesis effect and efficiency of speckle images and medical images. Fusing speckle images and medical images from a medical dataset to generate a synthetic dataset guarantees the accuracy of subsequent model training, improving the image reconstruction effect and generalization. A generative network enables image reconstruction based on local and global information in the image. Skip connections between the encoder and decoder allow the generator to capture rich contextual information at different levels, passing detailed information to the decoder and improving the decoder's accuracy in recovering image details. Simultaneously, the model has relatively few parameters and a simple structure, resulting in faster training and inference speeds and improved image reconstruction efficiency. A perceptual loss function, including content loss and adversarial loss, is adopted. By constraining the color distribution of the reconstructed image, the color distribution of the reconstructed image is made closer to the color distribution of the original high-resolution image, and the loss value is made more reflective of the perceptual similarity between images. This allows the generator network to produce higher discriminative values ​​and improves the model's interconnection effect. When reconstructing images, the white balance of the image can be achieved while removing honeycomb speckle. This achieves compatibility between thin fiber optic endoscopy and high-quality image acquisition, and improves the endoscopic image enhancement effect.

[0162] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0163] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image enhancement method characterized by, The application is applied to an endoscope image, comprising: Acquiring a medical data set and collecting a speckle image through a fiber endoscope; Generating a synthetic data set based on the speckle image and medical images in the medical data set, wherein the synthetic data set comprises synthetic images corresponding to each medical image in the medical data set; Conducting adversarial training on a deep learning model based on the synthetic data set and the medical data set to generate an image reconstruction model, and conducting image reconstruction on an endoscope image based on the image reconstruction model to obtain an image reconstruction result.

2. The method of claim 1, wherein, The collecting of the speckle image through the fiber endoscope comprises: Acquiring a pure white background image; Collecting an image through the fiber endoscope with the pure white background image as a target to obtain a speckle image.

3. The method of claim 1, wherein, The generating of the synthetic data set based on the speckle image and the medical images in the medical data set comprises: Pretreating the speckle image and the medical images in the medical data set; Multiplying and fusing the speckle image and the medical images to generate a synthetic image to obtain a synthetic data set.

4. The method of claim 1, wherein, The deep learning model comprises a generation network and a discrimination network, wherein the generation network is a convolutional neural network structure comprising an encoder and a decoder connected through a multi-scale jump connection, and the discrimination network is a convolutional neural network structure having the ability to discriminate the similarity of at least two input images and generate a discrimination result.

5. The method according to claims 1 to 4, characterized in that The adversarial training of the deep learning model based on the synthetic data set to generate an image reconstruction model comprises: Inputting a synthetic image in the synthetic data set into the generation network, conducting image reconstruction on the synthetic image based on the encoder and the decoder in the generation network to obtain a reconstruction image; Inputting the reconstruction image and the medical images in the medical data set into the discrimination network to conduct similarity discrimination to obtain a discrimination result; Updating parameters of the generation network and the discrimination network based on the discrimination result to complete the training of the generation network and the discrimination network to obtain a trained image reconstruction model. The image reconstruction of the synthetic image based on the encoder and the decoder in the generation network to obtain a reconstruction image comprises:

6. The method of claim 5, wherein, Conducting multi-level feature extraction on the synthetic image based on the encoder in the generation network to obtain image features; Conducting feature fusion on the image features based on the decoder in the generation network to obtain a reconstruction image. The updating of the parameters of the generation network and the discrimination network based on the discrimination result to complete the training of the generation network and the discrimination network to obtain a trained image reconstruction model comprises:

7. The method of claim 5, wherein, Determining a medical image corresponding to the synthetic image through the synthetic image source of the reconstruction image; Inputting the reconstruction image and the medical image into a loss function to calculate a loss value corresponding to the reconstruction image; ​ Determine whether the discrimination result is correct or not based on the correspondence between the reconstructed image and the medical image; if the discrimination result is correct, update the network parameters of the discrimination network based on the loss value; if the discrimination result is incorrect, update the network parameters of the generation network based on the loss value; and stop training until the loss function converges, and obtain the image reconstruction model.

8. An image enhancement device, characterized by The method comprises the steps of: An image acquisition module is configured to acquire a medical data set and collect speckle images through an optical fiber endoscope; A data set generation module is configured to generate a synthetic data set based on the speckle images and medical images in the medical data set, wherein the synthetic data set comprises synthetic images corresponding to each medical image in the medical data set; A training module is configured to perform adversarial training on a deep learning model based on the synthetic data set and the medical data set, generate an image reconstruction model, and perform image reconstruction on endoscope images based on the image reconstruction model to obtain an image reconstruction result.

9. An electronic device, comprising: The method comprises the steps of: At least one processor and at least one memory, Computer readable instructions are stored on the memory; The computer readable instructions are loaded and executed by the processor to enable the electronic device to implement the image enhancement method of any one of claims 1 to 7. The computer readable instructions are loaded and executed by the processor to implement the image enhancement method of any one of claims 1 to 7.

10. A storage medium having stored thereon computer readable instructions, characterized in that, ​

Images