Methods, equipment, storage media, and software products for denoising high-magnification microscope images

Abstract

This invention discloses a method, apparatus, storage medium, and program product for denoising high-magnification microscope images. The invention inputs a high-magnification noisy image acquired through a microscope into a trained denoising model to obtain a clear, denoised image. The training data for the denoising model includes training data of noisy images generated by a CycleGAN-based noise-adding model. This invention creatively uses a CycleGAN-based approach to generate high-magnification noisy images, overcoming the problem that in real-world scenarios, it is impossible to acquire images at both low and high magnification with the same image content.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and specifically relates to a noise reduction technique applied to high-magnification microscope images. Background Technology

[0002] In microscope imaging, at high magnification, noise is generated in the image due to resolution limitations in objective lens design. This noise typically manifests as texture blurring noise at the resolution limit, image sensor noise, and halo noise caused by optical scattering. Furthermore, the degree of each type of noise varies depending on the magnification. Therefore, improving image clarity at high magnification (i.e., reducing image noise) is a key issue in this field. Traditional methods generally use conventional digital image processing techniques to minimize noise levels, such as image denoising and image sharpening.

[0003] However, these traditional image processing methods offer limited improvement in image sharpness. In recent years, AI-based methods for high-magnification image denoising have been proposed, relying on deep learning models to enhance image quality. Specifically, this involves designing an AI model that takes a noisy image as input and outputs a clear image. However, for microscope scenes, there are currently no truly effective models of this type. The difficulty in developing a high-magnification denoising model for microscope scenes lies primarily in preparing the appropriate dataset—specifically, preparing a high-magnification noisy dataset of microscope images for training the deep learning model. Training a deep learning denoising model for microscope scenes requires data samples consisting of pairs of noisy and clear images of the same subject. In other words, during training, the model needs to be "told" what the original noisy image is and what the clear image to be generated is, and these two images must correspond one-to-one, meaning the image content is identical.

[0004] Previous methods for addressing the issue of creating one-to-one image pairs were based on clear images and involved manually adding noise to the images using image processing techniques to generate noisy images. Figure 1 As shown. The difficulty of this method lies in how to ensure that the manually added noise is consistent with the high-magnification noise encountered in reality. The key point is to ensure that the manually added noise is consistent with the high-magnification noise, so that the training set reflects the real situation and a high-performance denoising model can be trained.

[0005] In addition, in practical applications of microscopes, the magnification of the optical system is switched. That is, in actual use, a 500x optical system or a 3000x optical system may be used. Both of these magnifications can be called high magnifications, but the noise characteristics they produce are different. This is because the proportions of noise such as texture blur noise at the resolution limit, image sensor noise, and halo noise generated by optical scattering are different, and the overall noise pattern is also different. Summary of the Invention

[0006] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows: A method for denoising high-magnification microscope images involves inputting a high-magnification noisy image acquired by a microscope into a trained denoising model to obtain a clear image after denoising. The training data for the denoising model includes training data of noisy images generated by a CycleGAN-based noise-adding model. The CycleGAN-based noise-adding model comprises a first generator module, a second generator module, a noise-adding model magnification feature module, and a discriminator module. The first and second generator modules have the same image convolutional structure. The training process of the CycleGAN-based noise-adding model involves the first generator module inputting the original clear image acquired by the microscope at low magnification, while the noise-adding model magnification feature module acquires the magnification information of the original clear image and inputs it into the first generator module. The first generator module then outputs the generated noisy image. The second generator module takes the generated noisy image as input and outputs the generated clear image. It calculates the pixel loss and texture loss between the generated clear image and the original clear image, and optimizes the second generator module based on the pixel loss and texture loss. The discriminator module takes the generated noisy image and the original noisy image acquired by the microscope at high magnification as input. At the same time, the magnification feature module of the noise model acquires the magnification information of the original noisy image and inputs it into the discriminator module. The discriminator module distinguishes between the generated noisy image and the original noisy image and feeds back the discrimination result to the first generator module. This training process is repeated until the training completion condition is met to obtain the final first generator module. The clear image acquired by the microscope at low magnification is input into the final first generator module to generate noisy image training data for training the denoising model.

[0007] Furthermore, the pixel loss is calculated as follows:

[0008] in, L pixel For pixel loss function, H , W , CThe dimensions are, in order, the image's height, width, and number of channels. I ori ( i , j , c ) represents the pixel value of the original clear image. I rec ( i , j , c ) represents the pixel value of the generated clear image.

[0009] Furthermore, the method for calculating the texture loss is as follows: Set the texture feature vector of the original clear image. F ori : F ori = [ CON ori , COR ori , ENE ori , ENT ori ] Set the texture feature vector of the generated clear image. F rec : F rec = [ CON rec , COR rec , ENE rec , ENT rec ] in, CON , COR , ENE , ENT These represent contrast features, correlation features, energy features, and entropy features, respectively. calculate F ori and F rec The distance is the texture loss.

[0010] Furthermore, the training process of the denoising model is as follows: the training data of the noisy image generated by the final first generator module is mixed with the training data formed by manually adding noise according to a preset ratio to form the final training dataset. The final training dataset is input into the denoising model. At the same time, the magnification information of each image in the final training dataset is input into the denoising model through the denoising model magnification feature module to complete the training of the denoising model.

[0011] Secondly, the present invention also provides a computer device, which includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the above-described method for denoising high-magnification microscope images.

[0012] Furthermore, the present invention also provides a computer storage medium storing a computer program, which, when executed on a processor, implements the above-described method for denoising high-magnification microscope images.

[0013] Finally, the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the above-described method for denoising high-magnification microscope images.

[0014] The beneficial effects of adopting the above technical solution are as follows: This invention creatively utilizes a CycleGAN-based approach to generate high-magnification noisy images, overcoming the problem of acquiring images at both low and high magnification with the same content in real-world scenarios. Simultaneously, for microscopic scene images, a loss function constructed based on image texture features is used to constrain the texture similarity between the model-generated image and the real image, thereby increasing the quality of image texture generation. Furthermore, a magnification feature module is designed to incorporate magnification information; that is, the magnification information of the current optical system is input into the noise-adding model, and the model automatically generates images with corresponding noise features based on the magnification information. Similarly, a corresponding magnification feature module is also included in the noise-removing model to guide it in better removing noise. Attached Figure Description

[0015] Figure 1 These are schematic diagrams showing the image before and after manually adding noise; Figure 2 This is a diagram of the structure of the CycleGAN-based model with added noise constructed in this embodiment; Figure 3 This is a flowchart of the training process for the noisy model based on CycleGAN in this embodiment; Figure 4 This is a flowchart of the noise reduction model training process in this embodiment. Detailed Implementation

[0016] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0017] Microscopes can easily acquire clear images at low magnification and noisy images at high magnification. The challenge lies in maintaining consistency in image content between these two states, as the content of images captured at these two magnifications will inevitably differ. This problem is essentially due to the different fields of view; low magnification results in a large field of view, while high magnification results in a small field of view. Even cropping the low-magnification image (large field of view) to match the same content captured at high magnification cannot meet the requirements, because cropping the low-magnification image results in a very small area, which essentially becomes a low-resolution image. However, training a deep learning denoising model requires image pairs—a noisy image and a noise-free image, ensuring complete consistency in image content. Therefore, if there were a model that could transform a clear image into a noisy image at high magnification, the problem could be solved. Thus, this embodiment uses CycleGAN to address this issue. CycleGAN is an unsupervised model that can transform an image into another state, turning a clear image into a noisy image at high magnification. A batch of clear images and a batch of images with high-magnification noise (the image content is inconsistent) are collected. These two sets of images are defined as two image spaces. CycleGAN learns the transformation between these spaces in a cyclic learning manner, and finally achieves the transformation from clear space to noisy space.

[0018] like Figure 2 The diagram shows the noisy CycleGAN-based model structure designed in this embodiment. This structure includes texture loss and magnification feature modules. Training CycleGAN requires acquiring clear images at low magnification and noisy images at high magnification. Due to the different magnifications, the image content differs between these two scenarios. Figure 2In CycleGAN, the inputs are the "original sharp image" and the "original noisy image." The main structure includes a generator module G_AB for generating sharp to noisy images, a magnification feature module, a generator module G_BA for generating noisy to sharp images, and a discriminator. G_AB and G_BA have the same image convolutional structure, consisting of an encoder and a decoder. The encoder is a module based on a convolutional CNN layer and a downsampling cascade, while the decoder is a module based on a deconvolutional CNN layer and an upsampling cascade. G_AB takes a sharp image as input and outputs a noisy image, while G_BA takes a noisy image as input and outputs a sharp image. The discriminator is a module built on CNN and downsampling layers, ultimately outputting two nodes (values), distinguished as 1 and 0. The discriminator's function is to determine whether the input image comes from the output of G_AB or from an image acquired under high magnification.

[0019] Since the designed model distinguishes between different magnifications (500x to 3000x), the high-magnification (500x to 3000x) noisy images that need to be collected are acquired at different magnifications. For example, at six magnifications—500x, 1000x, 1500x, 2000x, 2500x, and 3000x—N images are acquired at each magnification, resulting in a total of N*6 high-magnification noisy images. Each image is labeled with the magnification at which it was captured. Therefore, the final dataset consists of N*6 high-magnification noisy images and M clear images. Generally, N*6 = M can be set.

[0020] like Figure 3 The diagram illustrates the training method for the noise-added model based on CycleGAN in this embodiment. First, images of different objects are acquired at low and high magnification; second, according to… Figure 2 The model inputs both the original clear image and the original noisy image into a designated module, while the magnification information of the original noisy image is input into the magnification feature module. The Discriminator is trained to distinguish whether an image originates from real-world capture or is generated by G_AB. The training signal of the Discriminator is fed back to G_AB, enabling G_AB to learn how to generate images that better "deceive" the Discriminator. The output of G_BA is compared with the input (original clear image) to calculate the image pixel loss and texture loss, and this loss is used to optimize G_BA. This training process is repeated continuously with training samples from the training set until training is complete. The image pixel loss is L1 loss, which is the difference between corresponding pixels in two images.

[0021] in, L pixel For pixel loss function, H , W , C The dimensions are, in order, the image's height, width, and number of channels. I ori ( i , j , c ) represents the pixel value of the original clear image. I rec ( i , j , c ) represents the pixel value of the generated clear image.

[0022] Texture loss is calculated by distributing the texture feature vectors of the original sharp image and the generated sharp image. The texture feature vector of the original sharp image is: F ori = [ CON ori , COR ori , ENE ori , ENT ori The texture feature vector of the generated sharp image: F rec = [ CON rec , COR rec , ENE rec , ENT rec That is, for any image, four features are extracted and combined to form the image's texture feature vector.

[0023] After the noisy model is trained, only the G_AB module is needed. The acquired clear images are continuously processed through G_AB to generate corresponding noisy images, thus completing the noisy image generation process and obtaining training samples with a one-to-one correspondence between clear and noisy images. These training samples can then be used to train the deep learning denoising model.

[0024] like Figure 4The diagram illustrates the denoising model training method in this embodiment. Assuming the training dataset obtained by the CycleGAN model is DatasetA, a training dataset named DatasetB is created using traditional image noise addition methods (Gaussian noise, white noise, fog noise, etc.). DatasetA and DatasetB are then mixed in a certain proportion to form the final Dataset, which is used to train the deep learning denoising model. The reason for this mixed data training is that the deep learning model can learn different types of noise, improving its denoising capabilities. Deep learning denoising models are generally based on CNN image conversion models. This embodiment adds a magnification feature module to the traditional CNN image conversion model, inputting magnification information into the model so that it can determine which magnification the noise originates from, thereby better removing the corresponding noise.

[0025] Furthermore, this embodiment does not use CycleGAN to perform image denoising because CycleGAN has a large computational load and cannot be applied in real time in actual use. Also, CycleGAN is prone to generating some traces during actual generation, which also affect the visual perception to some extent.

[0026] This embodiment also relates to a computer device, which includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the above-described contents.

[0027] This embodiment also relates to a computer storage medium storing a computer program, which, when executed on a processor, performs the above-described content.

[0028] This embodiment also relates to a computer program product, including a computer program or instructions, which, when executed by a processor, implement the above-mentioned content.

[0029] The embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for denoising high-magnification microscope images, characterized in that: High-magnification noisy images acquired by a microscope are input into a trained denoising model to obtain a clear image after denoising. The training data of the denoising model includes training data of noisy images generated by a CycleGAN-based noise-adding model. The CycleGAN-based noise-adding model includes a first generator module, a second generator module, a noise-adding model magnification feature module, and a discriminator module. The first generator module and the second generator module have the same image convolution structure. The training process of the CycleGAN-based noise-adding model is as follows: the first generator module receives the original clear image acquired by the microscope at low magnification, and simultaneously, the noise-adding model magnification feature module acquires the magnification information of the original clear image and inputs it into the first generator module. The first generator module outputs the generated noisy image, the second generator module inputs the generated noisy image and outputs the generated clear image, calculates the pixel loss and texture loss between the generated clear image and the original clear image, and optimizes the second generator module based on the pixel loss and texture loss, the discriminator module inputs the generated noisy image and the original noisy image acquired by the microscope at high magnification, and the noise model magnification feature module acquires the magnification information of the original noisy image and inputs it into the discriminator module, the discriminator module distinguishes between the generated noisy image and the original noisy image, and feeds back the discrimination result to the first generator module, and repeats this training process until the training completion condition is met, to obtain the final first generator module; Clear images acquired by the microscope at low magnification are input into the final first generator module to generate noisy image training data for training the denoised model.

2. The method for denoising high-magnification microscope images according to claim 1, characterized in that: The pixel loss is calculated as follows: ， in, L pixel For pixel loss function, H , W , C The dimensions are, in order, the image's height, width, and number of channels. I ori ( i , j , c ) represents the pixel value of the original clear image. I rec ( i , j , c ) represents the pixel value of the generated clear image.

3. The method for denoising high-magnification microscope images according to claim 1, characterized in that: The method for calculating the texture loss is as follows: Set the texture feature vector of the original clear image. F ori : F ori = [ CON ori , COR ori , ENE ori , ENT ori ] Set the texture feature vector of the generated clear image. F rec : F rec = [ CON rec , COR rec , ENE rec , ENT rec ] in, CON , COR , ENE , ENT These represent contrast features, correlation features, energy features, and entropy features, respectively. calculate F ori and F rec The distance is the texture loss.

4. The method for denoising high-magnification microscope images according to claim 1, characterized in that: The training process of the denoising model is as follows: the training data of the noisy image generated by the first generator module and the training data formed by manually adding noise are mixed according to a preset ratio to form the final training dataset. The final training dataset is input into the denoising model. At the same time, the magnification information of each image in the final training dataset is input into the denoising model through the denoising model magnification feature module to complete the training of the denoising model.

5. A computer device comprising a processor and a memory, the memory storing a computer program, the processor being configured to execute the computer program to implement the high-magnification microscope image noise reduction method as described in any one of claims 1-4.

6. A computer storage medium storing a computer program, which, when executed on a processor, implements the high-magnification microscope image noise reduction method as described in any one of claims 1-4.

7. A computer program product comprising a computer program or instructions that, when executed by a processor, implement the high-magnification microscope image noise reduction method as described in any one of claims 1-4.

Images