Image denoising method and device and cell segmentation method
By using a semi-supervised learning method based on neural network in retinal image denoising, the target image denoising model is trained, and the problem of poor denoising effect in the prior art is solved, and high-accuracy and efficient image denoising effect is achieved, which is suitable for high-quality processing of retinal images.
Patent Information
- Application Number
- CN202311700753.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-12
- Publication Date
- 2025-06-13
AI Technical Summary
The prior art considers fewer noise sources when denoising retinal images, resulting in large differences between the denoised images and the real images, and the denoised effect is poor, which cannot meet the needs of high-quality retinal images.
The target image denoising model is trained using a semi-supervised learning method based on a neural network, and trained using a clear image, a first noise image paired with it, and a second noise image without a paired clear image, the second noise image containing real noise and/or random noise to amplify the noise type and increase the generalization of the model.
It significantly improves the accuracy of the denoising image, enhances the image denoising effect, and can better preserve the details of the image, such as cell structure, which is conducive to downstream analysis, such as improving the accuracy of cell segmentation and meeting the needs of real-time online processing.
Smart Images
Figure CN120147168A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and more specifically, to an image denoising method, apparatus, and cell segmentation method. Background Art
[0002] Currently, retinal imaging has become a powerful technology for capturing high-resolution, subcellular-level images of the retina, enabling detailed visualization of cell structures, and providing valuable insights into retinal diseases such as retinitis pigmentosa and age-related macular degeneration, which is of great significance for the early screening of retinal and nervous system-related diseases. However, retinal images are usually vulnerable to inherent noise sources such as aberration blur, motion artifacts, and electronic noise, which may lead to a decline in the accuracy and interpretability of the captured data. Therefore, effectively denoising retinal images is a key step in improving their quality and facilitating downstream image analysis and clinical interpretation.
[0003] However, current image denoising methods consider fewer noise sources during the denoising process, resulting in a large difference between the denoised image and the real image, that is, the denoising effect is poor, the accuracy of the generated denoised image is low, and it cannot meet the requirements for high-quality retinal images. Summary of the Invention
[0004] Embodiments of the present application provide an image denoising method, apparatus, and cell segmentation method, which can improve the accuracy of the denoised image.
[0005] In view of this, embodiments of the present application provide an image denoising method, the method comprising:
[0006] Obtain an image to be denoised;
[0007] Input the image to be denoised into a target image denoising model to obtain a denoised image; wherein, the target image denoising model is trained in a semi-supervised manner based on a neural network, and the training images of the target image denoising model include a clear image, a first noise image paired with the clear image, and a second noise image without a paired clear image, and the second noise image is an image containing real noise or an image containing both real noise and random noise.
[0008] Optionally, the first noise image is an image in which random noise is synthesized in the clear image, and the real noise contained in the second noise image is the noise generated in the actually acquired image.
[0009] Optionally, the neural network is a generative adversarial network, and its training method comprises:
[0010] Input the first noise image into a generator to obtain a first denoised image;
[0011] Input the second noise image into a generator to obtain a second denoised image;
[0012] Construct a reconstruction loss function based on the first denoised image and the clear image;
[0013] Construct an adversarial loss function based on the first denoised image, the second denoised image, the clear image, and a discriminator;
[0014] Train to obtain a target image denoising model based on the minimization principle of the reconstruction loss function and the adversarial loss function.
[0015] Optionally, the training method further includes:
[0016] Construct a total variation loss function based on the first denoised image and the second denoised image;
[0017] Construct an information consistency loss function based on the first noise image and the first denoised image, and / or the second noise image and the second denoised image;
[0018] Train to obtain a target image denoising model based on the minimization principle of the reconstruction loss function, the adversarial loss function, the total variation loss function, and the information consistency loss function.
[0019] Optionally, the adversarial loss function includes a first adversarial loss function and a second adversarial loss function. The step of constructing an adversarial loss function based on the first denoised image, the second denoised image, the clear image, and a discriminator includes:
[0020] Input the first denoised image into the discriminator to obtain a first discrimination result;
[0021] Input the second denoised image into the discriminator to obtain a second discrimination result;
[0022] Input the clear image into the discriminator to obtain a third discrimination result;
[0023] Construct the first adversarial loss function of the generator based on the first discrimination result and the second discrimination result;
[0024] Construct the second adversarial loss function of the discriminator based on the first discrimination result, the second discrimination result, and the third discrimination result.
[0025] Optionally, the step of constructing an adversarial loss function based on the first denoised image, the second denoised image, the clear image, and a discriminator further includes:
[0026] Input the first noise image into the discriminator to obtain a fourth discrimination result;
[0027] Input the second noise image into a discriminator to obtain a fifth discrimination result;
[0028] Construct a second adversarial loss function of the discriminator according to the first discrimination result, the second discrimination result, the third discrimination result, the fourth discrimination result, and the fifth discrimination result.
[0029] Optionally, after obtaining the target image denoising model, the method further includes:
[0030] Obtain a verification image group, where the verification image group includes a verification image and a to-be-verified noise image corresponding to the verification image;
[0031] Input the to-be-verified noise image into the target image denoising model to obtain a to-be-verified denoised image;
[0032] Calculate the similarity between the to-be-verified denoised image and the verification image;
[0033] If the similarity is less than a preset threshold, re-execute the step of obtaining the to-be-trained image group.
[0034] An embodiment of the present application further provides a cell segmentation method, including:
[0035] Obtain a cell image; denoise the cell image using any one of the above-mentioned image denoising methods;
[0036] Perform cell segmentation on the denoised cell image.
[0037] An embodiment of the present application further provides an image denoising device, where the device includes:
[0038] An acquisition unit, configured to acquire an image to be denoised;
[0039] A denoising unit, configured to input the image to be denoised into a target image denoising model to obtain a denoised image;
[0040] A training unit, configured to train and obtain the target image denoising model in a semi-supervised manner based on a neural network; training images of the target image denoising model include a clear image, a first noise image paired with the clear image, and a second noise image without a paired clear image, and the second noise image is an image including real noise and / or random noise.
[0041] Optionally, the training unit is specifically configured to train and obtain a target image denoising model based on a generative adversarial network, including a generator, a discriminator, and a function construction unit;
[0042] The generator is used to input a first noisy image to obtain a first denoised image, and input a second noisy image to obtain a second denoised image;
[0043] The function construction unit is used to construct a reconstruction loss function according to the first denoised image and the clear image, and construct an adversarial loss function according to the first denoised image, the second denoised image, the clear image, and the discriminator;
[0044] The training unit is further used to train a target image denoising model based on the principle of minimizing the reconstruction loss function and the adversarial loss function.
[0045] An embodiment of the present application provides an image denoising method, and the method includes: obtaining an image to be denoised; inputting the image to be denoised into a target image denoising model to obtain a denoised image; wherein, the target image denoising model is trained in a semi-supervised manner based on a neural network, and the training images of the target image denoising model include clear images, first noisy images paired with the clear images, and second noisy images without paired clear images. The second noisy image is an image containing real noise or an image containing both real noise and random noise. It can be seen that in the present application, since the training images for training the target image denoising model not only include clear images and first noisy images paired with the clear images, but also include unpaired images containing real noise, the types of noise are expanded during the process of training the target denoising model, and the generalization of the target denoising model is increased.
[0046] The present invention uses a semi-supervised training method, combines self-supervision and unsupervision, and integrates unpaired low-quality images containing real-world data into an unsupervised adversarial learning framework, which can effectively process various types of noise, especially some noise sources that cannot be completely captured by synthetic blurring operations, such as electronic noise and eye movement in ophthalmic images, thereby being able to significantly improve the noise reduction ability and enhance the image quality. Compared with traditional image denoising methods and traditional deep learning models, the denoising results obtained by this semi-supervised denoising method have higher consistency with real high-quality images, improve the accuracy of denoising, and can better preserve the details of the image, such as cell structures, which is beneficial to assisting downstream analysis, such as improving the accuracy of cell segmentation.
[0047] Moreover, the processing speed of this method is faster and the efficiency is higher, enabling it to meet the real-time online image denoising processing requirements, which is crucial for research and clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on the provided drawings.
[0049] Figure 1 Schematic flowchart of an image denoising method provided by an embodiment of the present application;
[0050] Figure 2 Schematic flowchart of a training method for a target image denoising model provided by an embodiment of the present application;
[0051] Figure 3 Schematic diagram for comparing the denoising effects of an AO fundus image denoising model provided by an embodiment of the present application;
[0052] Figure 4 Schematic diagram for quantitatively comparing the denoising effects of an AO fundus image denoising model provided by an embodiment of the present application;
[0053] Figure 5 Schematic diagram for comparing the improvement in the effect of cell segmentation by an AO fundus image denoising model provided by an embodiment of the present application;
[0054] Figure 6 Schematic diagram of the structure of an image denoising device provided by an embodiment of the present application. Detailed implementation manners
[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
[0056] In the description and claims of this application and the above-mentioned drawings, terms such as "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.
[0057] Currently, image denoising methods based on conditional generative adversarial networks usually require paired high-quality and noisy images for training. Due to the deformation and focusing function of the eye lens, such paired real data is difficult to obtain in retinal imaging. Although attempts have been made to synthesize blurred images by manually introducing point spread function (PSF) convolution, since other noise sources in reality, such as electronic noise and eye movement, are not considered, there is a large difference between the denoised image and the real image, that is, the denoising effect is poor, the accuracy of the generated denoised image is low, and it cannot meet the requirements for high-quality retinal images.
[0058] Therefore, in view of the above problems, the embodiments of this application provide an image denoising method, device, and cell segmentation method, which can improve the accuracy of denoised images.
[0059] Please refer to Figure 1 , an image denoising method provided by the embodiments of this application includes the following steps.
[0060] S101. Obtain the image to be denoised.
[0061] In this embodiment, the image to be denoised can be obtained first. It can be understood that the image to be denoised can be an image including multiple or one type of noise, and it is an image that needs to be processed for image denoising. The image acquisition device is not limited, and can be various fundus cameras, or retinal imaging devices using adaptive optics (AO) technology. By compensating for eye aberrations, ultra-high-resolution retinal images can be obtained. This embodiment takes the AO retinal image containing noise captured in real time as an example.
[0062] S102. Input the image to be denoised into the target image denoising model to obtain a denoised image.
[0063] In this embodiment, after obtaining the image to be denoised, the image to be denoised can be input into the target image denoising model to obtain the denoised image. Among them, the target image denoising model is trained in a semi-supervised manner based on a neural network. The training images of the target image denoising model include clear images, first noise images paired with the clear images, and second noise images without paired clear images. The second noise images are images containing real noise or images containing both real noise and random noise. It can be understood that the clear image can be a clear and noise-free image, such as an AO retinal image, and the selection requirement is that the optical aberration is as small as possible, and there are no pictures such as eye movement, electronic or mechanical noise. Random noise can include random convolutional kernel Gaussian blur, random region convolutional kernel Gaussian blur, and radial blur in the whole image, etc. Real noise can include real noise sources that cause image blurring, such as eye movement, mechanical and electronic noise, etc. Since the training images for training the target image denoising model not only include clear images and first noise images paired with the clear images, but also include unpaired images containing real noise and random noise, that is, the types of noise are expanded during the process of training the target denoising model, and the generalization of the target denoising model is increased. Therefore, when the image to be denoised is input into the target image denoising model for image denoising, the target image denoising model can remove various noises in the image to be denoised as much as possible, thereby improving the accuracy of the finally generated denoised image and the image denoising effect.
[0064] In a possible implementation manner, the first noise image is an image in which random noise is synthesized in the clear image, and the real noise contained in the second noise image is the noise generated in the actually acquired image. It can be understood that the first noise picture can be an image containing random noise synthesized by a clear image and random noise, where the random noise includes random convolutional kernel Gaussian blur, random region convolutional kernel Gaussian blur, and radial blur in the whole image, etc. The second noise image can be an image containing noise from any source, such as an image containing real noise or an image containing both real noise and random noise. The real noise can include real noise sources that cause image blurring, such as eye movement, mechanical and electronic noise, etc.
[0065] In a possible implementation manner, the neural network can be a generative adversarial network. The first noise image can be input into the generator to obtain the first denoised image, and the second noise image can be input into the generator to obtain the second denoised image. A reconstruction loss function is constructed according to the first denoised image and the clear image. An adversarial loss function is constructed according to the first denoised image, the second denoised image, the clear image, and the discriminator. Based on the minimization principle of the reconstruction loss function and the adversarial loss function, the target image denoising model is trained.
[0066] It can be understood that in this embodiment, the target image denoising model can be trained in a semi-supervised manner based on a generative adversarial network, where the generative adversarial network includes a generator and a discriminator. The generator can be a neural network that takes a noisy image as input and outputs a denoised image, including an encoder and a decoder. The encoder can compress the spatial scale of the image to extract more global information, and the decoder can restore the signal encoded by the encoder to a denoised image. The encoder and decoder use multi-layer convolution and transposed convolution, and can be combined with a U-shaped neural network (Unet) and a residual neural network (ResNet) to achieve the best encoding and decoding effects. The discriminator can use three or more layers of convolution, and the convolution kernel can be selected as 1*1 or 3*3, and the stride can be selected as 1 or 2.
[0067] In the process of training the target image denoising model, the first noisy image and the second noisy image can be input into the generator respectively to obtain the corresponding first denoised image and second denoised image; after obtaining the first denoised image and the second denoised image, first, a reconstruction loss function can be constructed according to the first denoised image and the clear image. Specifically, the reconstruction loss function is one of the training objective functions of the target image denoising model, and the specific calculation method is the pixel brightness difference and structural difference between the clear image and the first denoised image. The specific calculation formula for the pixel brightness difference is:
[0068] L pixel = L1(G(synthetic LQ image), real HQ image);
[0069] where L1 is the mean absolute error, synthetic LQ image is the first noisy image, G(synthetic LQimage) is the first denoised image, and real HQ image is the clear image.
[0070] The specific calculation formula for the pixel brightness difference can also be:
[0071] L pixel = MSE(G(synthetic LQ image), real HQ image);
[0072] where MSE is the mean squared error, synthetic LQ image is the first noisy image, G(synthetic LQ image) is the first denoised image, and real HQ image is the clear image.
[0073] The structural difference can be measured by the error of the image features obtained by the VGG neural network between two pictures. The specific calculation formula is:
[0074] L perceptual = MSE(VGG(synthetic LQ image), VGG(real HQ image));
[0075] Wherein, VGG represents the image features after the Nth convolutional layer of VGG obtained by calculating after taking the image as the input of the VGG neural network, where N is usually 3 or 4, MSE is the mean square error, synthetic LQ image is the first noise image, and real HQ image is the real clear picture.
[0076] By minimizing the reconstruction loss function, the target image denoising model can learn to extract effective image information in a supervised manner.
[0077] Secondly, an adversarial loss function can be constructed according to the first denoised image, the second denoised image, the clear image, and the discriminator. The discriminator in the generative adversarial network can determine whether the denoised image generated by the generator is a clear image. Through the discrimination results of the discriminator, the generator and the discriminator can confront and learn from each other. The ultimate goal is to make the discriminator unable to determine whether the output result of the generator is real, so as to train the image denoising model with the best denoising effect. By minimizing the adversarial loss respectively in different stages of training the generator and the discriminator in the generative adversarial network, the target image denoising model can learn a more sensitive denoising adversarial network and learn to generate pictures similar to the real clear pictures, so as to achieve the purpose of denoising.
[0078] Specifically, the adversarial loss function includes a first adversarial loss function and a second adversarial loss function. The first denoised image can be input into the discriminator to obtain a first discrimination result; the second denoised image can be input into the discriminator to obtain a second discrimination result; the clear image can be input into the discriminator to obtain a third discrimination result; a first adversarial loss function of the generator is constructed according to the first discrimination result and the second discrimination result; a second adversarial loss function of the discriminator is constructed according to the first discrimination result, the second discrimination result, and the third discrimination result.
[0079] Among them, the calculation method of the first adversarial loss function is the cross-entropy of predicting the first denoised image as a clear image, and the specific calculation formula is:
[0080]
[0081] Wherein, MSE is the mean square error, D(G(synthetic LQ image) is the first discrimination result, and D(G(real LQ image) is the second discrimination result.
[0082] The calculation method of the second adversarial loss function is the sum of the cross-entropy of predicting the real clear image as a real image and the cross-entropy of predicting the first denoised image and the second denoised image as non-real images. The specific calculation formula is as follows:
[0083]
[0084] Among them, MSE is the mean square error, D(real HQ image,1) is the third discrimination result, D(G(synthetic LQimage),0) is the first discrimination result, and D(G(real LQ image),0) is the second discrimination result.
[0085] Finally, based on the minimization principle of the reconstruction loss function and the adversarial loss function, a target image denoising model can be trained. The adversarial learning based on this minimization principle enables the generative adversarial network model to process real-world noise by making the denoised image of the LQ input more similar to the real HQ image, so that the trained target image denoising model can have better image denoising effect, can remove various noises in the image to be denoised as much as possible, thereby improving the accuracy of the finally generated denoised image and improving the image denoising effect.
[0086] In a possible implementation manner, a total variation loss function can also be constructed according to the first denoised image and the second denoised image; based on the minimization principle of the reconstruction loss function, the adversarial loss function and the total variation loss function, a target image denoising model is trained.
[0087] It can be understood that a total variation loss function can be constructed according to the first denoised image and the second denoised image. The specific calculation method is to calculate the sum of the squares of the horizontal gradient and the vertical gradient of the first denoised image and the second denoised image. The specific calculation formula is as follows:
[0088]
[0089] Among them, mean is the mean of the total variation of all pixels of the denoised image, is the vertical discrete gradient of the denoised image, is the horizontal discrete gradient of the denoised image, synthetic LQ image is the first noise image, G(synthetic LQimage) is the first denoised image, real LQ image is the second noise image, and G(real LQ image) is the second denoised image. By minimizing the total variation loss function, the target image denoising model can learn to suppress the generation of noise points and image artifacts caused by the model or noise.
[0090] Based on the first noisy image and the first denoised image, and / or the second noisy image and the second denoised image, an information consistency loss function can be constructed to further improve the denoising performance of the trained target image denoising model. That is, an information consistency loss function is constructed based on the first noisy image and the first denoised image, and / or an information consistency loss function is constructed based on the second noisy image and the second denoised image. Adding the information consistency loss function can ensure the consistency of effective signals.
[0091] Finally, based on the minimization principle of the reconstruction loss function, the adversarial loss function, the total variation loss function, and the information consistency loss function, the target image denoising model can be trained, so that the trained target image denoising model can have a better image denoising effect, can remove various noises in the image to be denoised as much as possible, thereby improving the accuracy of the finally generated denoised image and the image denoising effect.
[0092] In a possible implementation manner, after the first denoised image is input into the discriminator to obtain the first discrimination result, and the second denoised image is input into the discriminator to obtain the second discrimination result, the first noisy image can also be input into the discriminator to obtain the fourth discrimination result, and the second noisy image is input into the discriminator to obtain the fifth discrimination result. Finally, the second adversarial loss function of the discriminator is constructed according to the first discrimination result, the second discrimination result, the third discrimination result, the fourth discrimination result, and the fifth discrimination result, so as to further improve the image denoising effect of the trained target image denoising model.
[0093] It can be understood that other neural networks can also be used for training. Those skilled in the art can replace the discriminator and the adversarial loss function unique to the above generative adversarial network with the architectures and parameters unique to other neural networks.
[0094] In a possible implementation manner, after the target image denoising model is trained, a validation image group can be obtained, where the validation image group includes a validation image and a to-be-validated noisy image corresponding to the validation image; the to-be-validated noisy image is input into the target image denoising model to obtain a to-be-validated denoised image; the similarity between the to-be-validated denoised image and the validation image is calculated; if the similarity is less than a preset threshold, the step of obtaining the to-be-trained image group is re-executed.
[0095] It is understandable that after the target image denoising model is obtained through training, in order to further verify the image denoising effect of the target image denoising model, a verification image group can be obtained, wherein the verification image in the verification image group is a clear noise-free image, and the noise image to be verified is a noisy image corresponding to the verification image. After the noise image to be verified is input into the target image denoising model to obtain the denoised image to be verified, the image denoising effect of the target image denoising model can be verified by calculating the similarity between the denoised image to be verified and the verification image. If the similarity is higher than a preset threshold, it means that the denoising effect of the target image denoising model meets the requirements; if the similarity is lower than the preset threshold, it means that the denoising effect of the target image denoising model does not meet the requirements and needs to be retrained. At this time, the step of obtaining the image group to be trained can be re-executed, and a new target image denoising model can be obtained by training by obtaining a new image group to be trained.
[0096] It can be seen from this that an embodiment of the present application provides an image denoising method. Since the training images used to train the target image denoising model include not only a clear image and a first noisy image paired with the clear image, but also an image containing real noise and random noise, that is, the types of noise are expanded in the process of training the target denoising model, and the generalization of the target denoising model is increased. Therefore, in the process of image denoising, the target image denoising model can remove various noises in the image to be denoised as much as possible, thereby improving the accuracy of the denoised image finally generated and improving the image denoising effect.
[0097] See also Figure 2 , an embodiment of the present application also provides a method for training a target image denoising model, which takes the removal of noise from AO retinal images as an example.
[0098] The denoising model used in the AO retinal image is based on a generative adversarial network (denoiseGAN), which utilizes both synthetic and real-world data to improve the denoising effect. In addition to traditional self-supervised learning of high-quality images and manually synthesized corresponding images, the AO retinal image denoising model also incorporates unpaired low-quality AO retinal images into the unsupervised adversarial learning framework. This combination helps the AO retinal image denoising model deal with real-world noise sources such as electronic noise and eye movements, which are difficult to capture with traditional image synthesis operations. Figure 3 The model training process shown mainly includes real clear picture 1, synthetic picture with random noise added 2, real noisy picture 3, denoising adversarial network generator 4, denoising picture 5, denoising adversarial network discriminator 6, reconstruction loss 7, total variation loss 8, and adversarial loss 9.
[0099] Among them, the real and clear picture 1 is a carefully selected AO fundus picture, and the selection requirements are that the optical aberration is as small as possible, and there is no eye movement, electronic, or mechanical noise. This picture is used to synthesize picture 2 with random noise and serves as the gold standard to assist the training of the supervised model. In the synthesized picture 2 with random noise, the random noise includes random convolution kernel Gaussian blur for the whole picture, random region convolution kernel Gaussian blur, and radial blur. The real noisy picture 3 is a picture with noise from any source, including aberration, eye movement, mechanical and electronic noise, etc., resulting in an unclear image, which is used as the guidance for unsupervised training. The denoising adversarial network generator 4 is a neural network that takes a noisy picture as input and outputs a denoised picture, including an encoder 4.1 and a decoder 4.2. The encoder compresses the spatial scale of the image to extract more global information, and the decoder restores the signal encoded by the encoder to a denoised image. The encoder and decoder use multi-layer convolution and deconvolution, and can be combined with the U-shaped neural network (Unet) and the residual neural network (ResNet) to achieve the best encoding and decoding effects.
[0100] The denoised image 5 is the input image after noise removal processed by the denoising adversarial network generator, which includes the first denoised image generated by inputting the image 2 with random noise into the generator 4 and the second denoised image output by inputting the real noisy image 3 into the generator 4. The denoising adversarial network discriminator 6 is a neural network used to improve the similarity between the denoised image 5 and the real clear image, and is only used for model training, not for model inference. The discriminator 6 can adopt three or more layers of convolution, the convolution kernel can be selected as 1*1 or 3*3, and the stride can be selected as 1 or 2. The reconstruction loss 7 is one of the model training objective functions, and the calculation method is the pixel brightness difference and structural difference between the original real clear image 1 and the corresponding denoised image 5; the pixel brightness difference is measured by L1 (mean absolute error) or L2 (mean square error), and the structural difference is measured by the error of the VGG neural network image features between the two images; by minimizing the reconstruction loss, the model learns to extract effective image information in a supervised manner. The total variation loss 8 is one of the model training objective functions, and the calculation method is the sum of the squares of the horizontal gradient and vertical gradient of the denoised image 5; by minimizing the total variation loss 8, the model learns to suppress the generation of noise and artifacts. The adversarial loss 9 is one of the model training objective functions. When training the denoising adversarial network generator 4, the calculation method is the cross-entropy that the denoising adversarial network 6 predicts the denoised image 5 as a real image; when training the denoising adversarial network discriminator 6, the calculation method is the sum of the mean square error or cross-entropy that the denoising adversarial network 6 predicts the real clear image 1 as a real image, and the mean square error or cross-entropy that the denoising adversarial network 6 predicts the denoised image 5 as a non-real image. By minimizing the adversarial loss 9 at different stages of training the adversarial network generator 4 and the discriminator 6 respectively, the model learns a more sensitive denoising adversarial network discriminator 6 and learns to generate an image 5 similar to the real clear image 1, so as to achieve the purpose of denoising.
[0101] Figure 3 This is the effect diagram of the AO retinal image denoising model provided by the embodiment of the present application on the independent test set, as well as the comparison with other methods. From left to right are: the original clear AO fundus image, the blurred image obtained by artificially adding noise, the AO retinal image denoising model, wavelet function filtering, winer filtering, and the denoising results of cGAN for the blurred image. The original clear AO fundus image is used as the gold standard to measure the result differences generated by the gold standard and various denoising methods, and to evaluate and quantify the denoising effect.
[0102] Figure 4The image denoising effect of the AO retinal image denoising model provided by the embodiments of this application is quantified on an independent test set. The measurement criteria include the mean square error (MSE), structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), and blind image quality index (BIQI) between the denoised image and the original clear image. The AO retinal image denoising model provided by the embodiments of this application significantly outperforms traditional methods in all indicators.
[0103] Figure 5 For the improvement of the downstream cell segmentation performance by the AO retinal image denoising model provided by the embodiments of this application, from left to right are: gold standard labeled cell segmentation, blurred image without using the AO retinal image denoising model, cell segmentation results based on the blurred image without using the AO retinal image denoising model, image processed by the AO retinal image denoising model, and cell segmentation results based on the image denoised by the AO retinal image denoising model. When not denoised by the AO retinal image denoising model, the precision of cell segmentation is 0.9943±0.0019, and the recall is 0.5776±0.1038; after denoising by the AO retinal image denoising model, the precision is improved to 0.9951±0.0014, and the recall is improved to 0.6995±0.0960.
[0104] In terms of computational efficiency, the semi-supervised neural network model generator adopted by this method contains 15,798,723 parameters, which is approximately one-fourth of the previous state-of-the-art cGAN model, whose generator contains 54,676,097 parameters. When processing 256*256 pixel images on an Intel 6248R CPU, the semi-supervised neural network takes 0.14 seconds, while the originally published cGAN claims a speed of 1 second per image. The high efficiency of this method enables it to meet the real-time online processing requirements of AO retinal images in a clinical environment and has broad application prospects in various image processing in other fields.
[0105] Experimental results show that the semi-supervised neural network can restore high-quality images, retain cell structures, and enhance contrast, outperforming traditional image denoising methods and the state-of-the-art cGAN model. In addition, the contribution of the semi-supervised neural network is not limited to image restoration. It also helps downstream analysis and improves the accuracy of cell segmentation. The high computational efficiency of the semi-supervised neural network further enhances its practicality, making it suitable for real-time online image denoising processing. Generally speaking, the semi-supervised neural network has made valuable progress in image denoising and has potential application value in research and clinical practice.
[0106] It can be seen that the training method of a target image denoising model provided by the embodiments of the present application is based on a generative adversarial network, uses synthetic and real-world data, and considers different types of noise, including random noise, aberration, eye movement, mechanical noise, etc., to ensure the comprehensiveness of denoising. On an independent test set, the present application shows significantly better results than traditional methods. Through quantitative evaluations of indicators such as mean squared error, structural similarity, peak signal-to-noise ratio, and reference-free image quality assessment, it is proved that the denoising performance of the target image denoising model is excellent, the signal-to-noise ratio is improved, and at the same time the effective information of the original image is retained. The present application also improves the performance of downstream cell segmentation tasks. The images processed by the model denoising perform better in cell segmentation, and both the precision rate and the recall rate are significantly improved, which is of great significance for accurate cell density analysis and disease detection in clinical practice.
[0107] The embodiments of the present application also provide a cell segmentation method, which specifically includes the following steps:
[0108] Obtain a cell image; denoise the cell image using any one of the image denoising methods described above;
[0109] Perform cell segmentation on the denoised cell image.
[0110] In this embodiment, by using any one of the image denoising methods described above to denoise the obtained cell image and then performing cell segmentation on the denoised cell image, the performance of cell segmentation can be greatly improved. The cell images processed by denoising perform better in cell segmentation, and both the precision rate and the recall rate are significantly improved, which is of great significance for accurate cell density analysis and disease detection in clinical practice. Moreover, the semi-supervised neural network model shows the ability to enhance the separation of adjacent cells and overcomes the challenges brought by noise, which may hinder the accurate determination of cell centers by cell segmentation algorithms. Therefore, using the semi-supervised neural network model helps to identify more cells that may be missed during the detection process when dealing with blurred images and ensures its detection accuracy.
[0111] Please refer to Figure 6 , the embodiments of the present application also provide an image denoising device, and the device includes:
[0112] An acquisition unit 601, configured to acquire an image to be denoised;
[0113] A denoising unit 602, configured to input the image to be denoised into a target image denoising model to obtain a denoised image;
[0114] A training unit 603 for training the target image denoising model in a semi-supervised manner based on a neural network; the training images of the target image denoising model include a clear image, a first noisy image paired with the clear image, and a second noisy image without a paired clear image, and the second noisy image is an image containing real noise and / or random noise.
[0115] Optionally, the training unit 603 is specifically configured to train a target image denoising model based on a generative adversarial network, including a generator, a discriminator, and a function construction unit;
[0116] The generator is configured to input a first noisy image to obtain a first denoised image, and input a second noisy image to obtain a second denoised image;
[0117] The function construction unit is configured to construct a reconstruction loss function according to the first denoised image and the clear image, and construct an adversarial loss function according to the first denoised image, the second denoised image, the clear image, and the discriminator;
[0118] The training unit is further configured to train a target image denoising model based on the principle of minimizing the reconstruction loss function and the adversarial loss function. Optionally, the training unit is further configured to construct a total variation loss function according to the first denoised image and the second denoised image;
[0119] The training unit is further configured to train a target image denoising model based on the principle of minimizing the reconstruction loss function, the total variation loss function, and the adversarial loss function.
[0120] Optionally, construct an information consistency loss function according to the first noisy image and the first denoised image, and / or the second noisy image and the second denoised image;
[0121] The training unit is further configured to train a target image denoising model based on the principle of minimizing the reconstruction loss function, the total variation loss function, the adversarial loss function, and the information consistency loss function.
[0122] Optionally, the adversarial loss function includes a first adversarial loss function and a second adversarial loss function, and the training unit is specifically configured to:
[0123] Input the first denoised image into the discriminator to obtain a first discrimination result;
[0124] Input the second denoised image into the discriminator to obtain a second discrimination result;
[0125] Input the clear image into the discriminator to obtain a third discrimination result;
[0126] Construct a first adversarial loss function of the generator according to the first discrimination result and the second discrimination result;
[0127] Construct a second adversarial loss function of the discriminator according to the first discrimination result, the second discrimination result, and the third discrimination result.
[0128] Optionally, input the first noise image into the discriminator to obtain a fourth discrimination result;
[0129] Input the second noise image into the discriminator to obtain a fifth discrimination result;
[0130] Construct a second adversarial loss function of the discriminator according to the first discrimination result, the second discrimination result, the third discrimination result, the fourth discrimination result, and the fifth discrimination result.
[0131] Optionally, the device further includes:
[0132] The verification acquisition unit is further configured to acquire a verification image group, where the verification image group includes a verification image and a to-be-verified noise image corresponding to the verification image;
[0133] The verification unit is configured to input the to-be-verified noise image into the target image denoising model to obtain a to-be-verified denoised image;
[0134] The calculation unit is configured to calculate the similarity between the to-be-verified denoised image and the verification image;
[0135] The execution unit is configured to, if the similarity is less than a preset threshold, re-execute the step of acquiring the to-be-trained image group.
[0136] It can be seen that the embodiment of the present application provides an image denoising device. Since the training images for training the target image denoising model include not only clear images and the first noise images paired with the clear images, but also images containing real noise and random noise, that is, the types of noise are expanded during the process of training the target denoising model, and the generalization of the target denoising model is increased. Therefore, in the process of image denoising, the target image denoising model can remove various noises in the to-be-denoised image as much as possible, thereby improving the accuracy of the finally generated denoised image and the image denoising effect.
[0137] The embodiment of the present application further provides a computer device, including: a memory, a processor, and a bus system;
[0138] Wherein, the memory is used to store programs;
[0139] The processor is configured to execute the programs in the memory to implement any one of the above image denoising methods;
[0140] The bus system is used to connect the memory and the processor, so that the memory and the processor can communicate with each other.
[0141] An embodiment of the present application further provides a computer-readable storage medium storing instructions, which, when running on a computer, cause the computer to execute any one of the image denoising methods described above.
[0142] Finally, it should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.
[0143] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An image denoising method, characterized in that, the method includes: obtaining an image to be denoised; inputting the image to be denoised into a target image denoising model to obtain a denoised image; wherein, the target image denoising model is trained in a semi-supervised manner based on a neural network, and the training images of the target image denoising model include clear images, first noise images paired with the clear images, and second noise images without paired clear images, and the second noise images are images containing real noise or images containing both real noise and random noise.
2. The method according to claim 1, characterized in that, the first noise image is an image in which random noise is synthesized in a clear image, and the real noise contained in the second noise image is the noise generated in an actually acquired image.
3. The method according to claim 1, characterized in that, the neural network is a generative adversarial network, and its training method includes: inputting the first noise image into a generator to obtain a first denoised image; inputting the second noise image into the generator to obtain a second denoised image; constructing a reconstruction loss function according to the first denoised image and the clear image; constructing an adversarial loss function according to the first denoised image, the second denoised image, the clear image, and a discriminator; training a target image denoising model based on the principle of minimizing the reconstruction loss function and the adversarial loss function.
4. The method according to claim 3, characterized in that, the training method further includes: constructing a total variation loss function according to the first denoised image and the second denoised image; constructing an information consistency loss function according to the first noise image and the first denoised image, and / or the second noise image and the second denoised image; training a target image denoising model based on the principle of minimizing the reconstruction loss function, the adversarial loss function, the total variation loss function, and the information consistency loss function.
5. The method according to claim 3, characterized in that, the adversarial loss function includes a first adversarial loss function and a second adversarial loss function, and constructing an adversarial loss function according to the first denoised image, the second denoised image, the clear image, and a discriminator includes: inputting the first denoised image into the discriminator to obtain a first discrimination result; inputting the second denoised image into the discriminator to obtain a second discrimination result; inputting the clear image into the discriminator to obtain a third discrimination result; constructing the first adversarial loss function of the generator according to the first discrimination result and the second discrimination result; constructing the second adversarial loss function of the discriminator according to the first discrimination result, the second discrimination result, and the third discrimination result.
6. The method according to claim 5, characterized in that, constructing an adversarial loss function according to the first denoised image, the second denoised image, the clear image, and a discriminator further includes: inputting the first noise image into the discriminator to obtain a fourth discrimination result; inputting the second noise image into the discriminator to obtain a fifth discrimination result; Construct a second adversarial loss function of the discriminator according to the first discrimination result, the second discrimination result, the third discrimination result, the fourth discrimination result, and the fifth discrimination result.
7. The method according to claim 1, wherein, after obtaining the target image denoising model, the method further includes: obtaining a verification image group, where the verification image group includes a verification image and a to-be-verified noise image corresponding to the verification image; inputting the to-be-verified noise image into the target image denoising model to obtain a to-be-verified denoised image; calculating the similarity between the to-be-verified denoised image and the verification image; if the similarity is less than a preset threshold, re-execute the step of obtaining the to-be-trained image group.
8. A cell segmentation method, wherein, includes: obtaining a cell image; denoising the cell image using the image denoising method according to any one of claims 1-7; performing cell segmentation on the denoised cell image.
9. An image denoising device, wherein, the device includes: an obtaining unit, configured to obtain an image to be denoised; a denoising unit, configured to input the image to be denoised into a target image denoising model to obtain a denoised image; a training unit, configured to train the target image denoising model in a semi-supervised manner based on a neural network; the training images of the target image denoising model include a clear image, a first noise image paired with the clear image, and a second noise image without a paired clear image, and the second noise image is an image containing real noise and / or random noise.
10. The device according to claim 9, wherein, the training unit is specifically configured to train a target image denoising model based on a generative adversarial network, including a generator, a discriminator, and a function construction unit; the generator is configured to input a first noise image to obtain a first denoised image, and input a second noise image to obtain a second denoised image; the function construction unit is configured to construct a reconstruction loss function according to the first denoised image and the clear image, and construct an adversarial loss function according to the first denoised image, the second denoised image, the clear image, and the discriminator; the training unit is further configured to train the target image denoising model based on the principle of minimizing the reconstruction loss function and the adversarial loss function.