Eye ground color photo repairing method, device and system based on diffusion model, medium and program product

Abstract

The invention provides a fundus color photograph repairing method, system and device based on a diffusion model, a medium and a program product, and the method comprises the steps: converting a plurality of color images into a CMYK format, quantifying a black plane to establish a dynamic threshold value, carrying out the morphological operation through employing a binary mask, and carrying out the restoration of a fundus color photograph. Selecting the maximum area in each image as a mask to identify the most significant shadow area, and constructing a mask library according to the most significant shadow area; adding noise to an original image based on the mask library; constructing a noise prediction model based on the original image and the noise image; based on the noise prediction model, training a U-Net model to obtain a diffusion model used for recovering an original image from a noise image; and performing model reasoning by using the trained diffusion model so as to reconstruct the image with the shadow into a clean image without loss. According to the technical scheme, the precision and efficiency of image processing are improved, the generalization ability and robustness of the model are enhanced, and the details and precision of the image can be well restored for the situation that an existing algorithm (such as a generative adversarial network or an automatic encoder) is limited in effectiveness when the image suffers from major detail loss.

Description

Technical Field

[0001] The present application relates to the field of image reconstruction technology, and in particular to a method, system, device, medium and program product for restoring fundus color photographs based on a diffusion model. Background Art

[0002] In the field of ophthalmology, many diagnostic tasks rely on the analysis of fundus images. However, the effectiveness of the diagnostic process is often limited by the quality of retinal images. Low-quality fundus images not only pose a huge challenge to ophthalmologists, affecting the efficiency and accuracy of their diagnosis, but also seriously weaken the performance of automatic retinal evaluation systems. This is particularly evident when there are significant defects in the image due to lens opacities, severe cataracts, vitreous opacities or other obstacles. Due to these factors, difficulties may be encountered when making a diagnosis, resulting in an increased risk of misdiagnosis or missed diagnosis. Therefore, improving the quality of fundus images is crucial to improving the reliability of diagnosis and the effectiveness of automatic evaluation systems.

[0003] High-quality fundus images are critical, whether manually annotated by medical professionals or analyzed using automated methods such as artificial intelligence. The integrity of these images directly affects the accuracy and reliability of diagnosis. Convincing diagnoses rely on clear and detailed image data so that doctors can accurately identify and assess the lesion area. In recent advances in this field, researchers have made extensive use of deep learning techniques such as generative adversarial networks (GANs) and autoencoders (AEs) to restore or enhance fundus images. These techniques have shown great potential in image processing and enhancement, especially in style transfer and global brightness enhancement. However, the utility of these methods remains limited when images suffer from significant detail loss, such as the presence of extensive shadow areas. Although GANs and AEs are able to improve image quality in some cases, they still face challenges when processing severely damaged images and cannot fully restore the details and accuracy of the images. Summary of the invention

[0004] In view of the shortcomings of the prior art described above, the purpose of the present application is to provide a method, system, device, medium and program product for fundus color photography restoration based on a diffusion model, so as to solve the technical problem that the prior art cannot fully restore the details and accuracy of the image when processing severely damaged images.

[0005] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a method for fundus color photography restoration based on a diffusion model, comprising: converting a number of color images into CMYK format, quantizing the black plane to establish a dynamic threshold, applying a binary mask to perform morphological operations, selecting the largest area in each image as a mask to identify the most significant shadow area, and constructing a mask library based on this; adding noise to the original image based on the mask library; constructing a noise prediction model based on the original image and the noisy image; based on the noise prediction model, training a U-Net model to obtain a diffusion model for recovering the original image from the noisy image; and using the trained diffusion model to perform model inference to reconstruct the shadowed image into a clean image without loss.

[0006] In some embodiments of the first aspect of the present application, the method of converting a plurality of color images into CMYK format, quantifying the black plane to establish a dynamic threshold, applying a binary mask to perform morphological operations, selecting the largest area in each image as a mask to identify the most significant shadow area, and constructing a mask library accordingly includes: converting a plurality of color images into CMYK format, performing quantitative analysis on the black plane in the CMYK format, and establishing two dynamic thresholds for distinguishing different areas in the image based on the quantitative analysis results, so as to identify and separate parts of the area of ​​interest; based on the two dynamic thresholds, applying a binary mask to perform morphological reconstruction and closing operations on the image; selecting the most significant or largest area from the image after the morphological reconstruction and closing operations as the final mask for characterizing the most significant shadow area in the image, so as to generate a mask library.

[0007] In some embodiments of the first aspect of the present application, the image is morphologically reconstructed and closed based on two dynamic thresholds using a binary mask, which includes: applying a binary mask to the image using the dynamic threshold to convert the image into a binary image; performing morphological reconstruction and closing operations on the binary image, the morphological reconstruction being used to restore the destroyed structure, and the closing operation being used to eliminate small holes and connect disconnected parts.

[0008] In some embodiments of the first aspect of the present application, noise is added to the original image based on the mask library; a noise prediction model is constructed based on the original image and the noise image; based on the noise prediction model, a U-Net model is trained to obtain a diffusion model for recovering the original image from the noise image, which includes: randomly selecting a mask from the mask library and applying it to the original image to generate a corresponding mask image, and adding noise only in the mask area of ​​the mask image; constructing a noise prediction model; the noise prediction model is used to describe the mapping relationship between the noise image and the original image; constructing a U-Net model, and using the noise prediction model to define a loss function, and adjusting the parameters of the U-Net model through a gradient descent algorithm until the loss function is minimized.

[0009] In some embodiments of the first aspect of the present application, the process of constructing the repaired image with severe black shadows includes: given a noisy image X T , the goal is to reconstruct the image to its original state X0. The diffusion process of forward noise addition is:

[0010] Among them, q(X t |X t-1 ) indicates that given the previous state X t-1 Lower state X t The conditional probability distribution of And the covariance matrix is ​​β t Normal distribution of I, represented by the symbol N; mean Indicates the current state X t is the previous state X t-1 A scaled version of Covariance matrix β t I, β t is a scalar representing the noise level or uncertainty of the process, and I is the identity matrix, indicating that the covariance matrix is ​​diagonal and the variance of each dimension is β t .

[0011] In some embodiments of the first aspect of the present application, the learning objective of the U-Net model is: The loss function of the U-Net model is: in, Indicates the unmasked area X cond and the noisy image X T The expected value of E (∈，α) represents the expected value of noise ∈ and parameter α; f θ (X cond , X T ,γ) is the output of the U-Net model, indicating that in a given unmasked area X cond and the noisy image X T In the case of , the model predicts the noise, we define γ represents the proportion of data retained; L represents the difference between the predicted noise and the real noise ∈ p The p-th power of the norm, which is used to measure the difference between the noise predicted by the model and the actual noise.

[0012] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a fundus color photo restoration system based on a diffusion model, including: a mask library construction module, used to convert a number of color images into CMYK format, quantize the black plane to establish a dynamic threshold, apply a binary mask to perform morphological operations, select the largest area in each image as a mask to identify the most significant shadow area, and construct a mask library based on this; a model training module, used to add noise to the original image based on the mask library; construct a noise prediction model based on the original image and the noisy image; based on the noise prediction model, train a U-Net model to obtain a diffusion model for recovering the original image from the noisy image; a model inference module, used to perform model inference using the trained diffusion model to reconstruct the shadowed image into a clean image without loss.

[0013] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for restoring fundus color photography based on a diffusion model is implemented.

[0014] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the fundus color photography restoration method based on the diffusion model.

[0015] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides a computer device / equipment / system / electronic terminal, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the diffusion model-based fundus color photography restoration method.

[0016] As described above, the diffusion model-based fundus color photograph restoration method, system, device, medium and program product of the present application have the following beneficial effects: the technical solution of the present application not only improves the accuracy and efficiency of image processing, but also enhances the generalization ability and robustness of the model. For existing algorithms (such as generative adversarial networks or autoencoders) that have limited effectiveness when faced with significant detail loss in images, the present application can well restore the details and accuracy of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a flow chart of a method for restoring fundus color photography based on a diffusion model in one embodiment of the present application.

[0018] Figure 2 It is a schematic diagram of the sub-step flow chart of step S1 in one embodiment of the present application.

[0019] Figure 3 Shown is a schematic diagram of the model training process and model reasoning process in one embodiment of the present application.

[0020] Figure 4 A schematic diagram showing the entire process of reconstructing a fundus image in one embodiment of the present application is shown.

[0021] Figure 5 Shown is a structural schematic diagram of a fundus color photography restoration system based on a diffusion model in one embodiment of the present application.

[0022] Figure 6 Shown is a schematic diagram of the structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0024] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0025] <1> Generative Adversarial Network (GAN), a deep learning model, was proposed by Ian Goodfellow et al. in 2014. It consists of two parts: the generator and the discriminator. The generator generates fake data that is as close to the real data as possible, while the discriminator distinguishes between the real data and the fake data generated by the generator. The two networks compete with each other during the training process. The generator constantly learns how to generate more realistic data, while the discriminator constantly learns how to identify fake data more accurately. When the two reach a balance, the quality of the data generated by the generator is usually very high and can be used for applications such as image generation, style transfer, and data enhancement.

[0026] <2> Autoencoder (AE), a neural network for data compression and feature learning, is mainly used for data compression and feature learning. It encodes the input data into a low-dimensional representation through an encoding process, and then reconstructs this low-dimensional representation back to the original data through a decoding process. The purpose of the autoencoder is to minimize the difference between the input data and the reconstructed data. During the encoding process, the autoencoder learns a compressed representation of the input data, which can be used for tasks such as data dimensionality reduction and feature extraction. In addition, the autoencoder can also be used for applications such as anomaly detection and denoising.

[0027] <3> CMYK format: a color mode, the full name is Cyan, Magenta, Yellow and Key (the "Key" here refers to the last link, that is, "K" stands for black).

[0028] <4> U-Net structure: A deep learning network architecture that is often used for image segmentation tasks, especially in the field of medical image processing. The structure of U-Net is "U" shaped, consisting of a contraction path (encoder) and an expansion path (decoder).

[0029] <5> Binary mask: A special image that contains only two pixel values, usually 0 and 1 or black and white. In image processing, binary masks are used to indicate the presence or absence of certain areas in an image, or to specify that certain areas should be processed or ignored. For example, in image segmentation, a binary mask can be used to indicate foreground areas (pixels with a value of 1) and background areas (pixels with a value of 0).

[0030] <6> Diffusion Models is a generative model that transforms data into a Gaussian distribution by gradually adding noise, and then gradually removes the noise through a reverse process to restore the original data.

[0031] In order to solve the technical problems described above, the present application introduces a new method that uses a diffusion generation model to restore low-quality fundus color images, especially images with shadows. It should be understood that the method provided in the embodiment of the present application can be applied to other types of medical images in the medical field, such as X-ray images, CT (computed tomography) images, MRI (magnetic resonance imaging) images, SPECT (single photon emission computed tomography) and PET (positron emission computed tomography) images, ultrasound images, pathological slice images, endoscopic images, etc. In addition, in addition to being applied to the medical field, it can actually be used in industrial detection, video surveillance, military defense, aerospace, automatic driving and other fields, and the present application does not limit this.

[0032] Figure 1 The flowchart of a method for restoring fundus color photos based on a diffusion model in an embodiment of the present invention is shown. The method in this embodiment mainly includes the following steps:

[0033] Step S1: convert several color images into CMYK format, quantize the black plane to establish a dynamic threshold, apply a binary mask to perform morphological operations, select the largest area in each image as a mask to identify the most significant shadow area, and construct a mask library based on this.

[0034] In the embodiment of the present application, the specific implementation process of step S11 can be divided into: Figure 2 The multiple steps shown:

[0035] Step S1a: converting a number of color images into CMYK format, performing quantitative analysis on the black plane in the CMYK format, and establishing two dynamic thresholds for distinguishing different areas in the image according to the quantitative analysis results, so as to identify and separate the part of the area of ​​interest.

[0036] It should be understood that the purpose of converting a color image to CMYK format is to be able to better separate and process the different color components of the image. CMYK format is a color mode, the full name is Cyan, Magenta, Yellow and Key (Key here refers to the last link, that is, "K" stands for black). In CMYK format, colors are created by different combinations of these four basic inks, and each ink can have a concentration from 0% (no ink at all) to 100% (fully saturated). By changing the concentration of each ink, millions of different colors can be produced.

[0037] In some examples, the quantitative analysis of the black plane in the CMYK format includes calculating the average intensity and standard deviation of the black plane component. These two statistics can describe the central tendency and dispersion of the color distribution on the black plane. Among them, the average intensity represents the average level of all pixel values ​​on the black plane. It is the center point of the color distribution and can be used as a benchmark for color depth. In image processing, the average intensity can be used to estimate the overall darkness of the image. The standard deviation is a statistic that measures the degree of dispersion of a set of values. It indicates the degree of deviation between the values ​​in the data set and the average value. In image processing, the standard deviation can reflect the range of variation of color depth on the black plane.

[0038] By calculating the average intensity and standard deviation of the black plane component, two dynamic thresholds, low threshold and high threshold, can be established. The low threshold is usually set to the average intensity minus several times the standard deviation, and the high threshold is usually set to the average intensity plus several times the standard deviation. The area below the low threshold is considered to be the lighter part, corresponding to the bright area or background noise in the image; the area above the high threshold is considered to be the darker part, corresponding to the shadow or area of ​​interest in the image. Here, "several times" of the standard deviation as the offset of the threshold can be determined according to the specific application and image characteristics. Such a design can make the threshold not fixed, but dynamically calculated according to the characteristics of the image data itself, so it is more flexible and adaptable.

[0039] Step S1b: Based on two dynamic thresholds, a binary mask is applied to perform morphological reconstruction and closing operations on the image.

[0040] First, a binary mask is applied to the image using the dynamic threshold established in the previous step to convert the image into a binary image.

[0041] Specifically, for each pixel in the image, a mask value is assigned based on the relationship between its intensity value and the two thresholds. Usually, if the pixel value is above the upper threshold or below the lower threshold, the mask value is set to 1 (or white), indicating that the pixel belongs to the region of interest; otherwise, the mask value is set to 0 (or black), indicating that the pixel does not belong to the region of interest. As needed, the thresholds can be fine-tuned to ensure that the key features in the image are correctly captured. For each pixel in the original image, the mask value is used to determine the value of the pixel in the binary image. If the mask value is 1, the original pixel value is retained or converted to white; if the mask value is 0, the original pixel value is converted to black. In addition, some post-processing operations can be performed, such as removing small noise points, filling small holes, smoothing boundaries, etc., to improve the quality of the binary image.

[0042] Subsequently, the binary image is subjected to morphological reconstruction and closing operations. Morphological reconstruction helps to restore structures destroyed by noise or other interference, while closing operations eliminate small holes and connect disconnected parts to ensure the continuity and integrity of the detection results. In medical image processing, morphological reconstruction and closing operations are often used to improve image quality, eliminate noise, and prepare images for further analysis, such as feature extraction, segmentation, and recognition. Through these operations, the accuracy and reliability of image analysis can be improved.

[0043] Morphological reconstruction is a morphological operation used to extract specific structures in an image and eliminate noise and other interference that do not belong to these structures. It is often used to separate different components in an image and can restore destroyed structures. Morphological reconstruction is usually implemented using a combination of erosion and dilation: the erosion operation reduces the size of objects in the image and removes parts of the edge of the object. It compares each pixel in the image with its structural element. Only when the structural element is completely contained in the corresponding area of ​​the original image, the pixel will be retained. The dilation operation increases the size of objects in the image and fills the holes inside the object. It prevents the center of the structural element from being on each pixel of the image. If the structural element overlaps with any part of the original image, the pixel is set to the foreground (usually white or 1).

[0044] Morphological Closing is another morphological operation used to fill small holes and cracks inside objects in an image, as well as to connect adjacent objects. Closing is actually a combination of dilation and erosion, and the steps are as follows: first, dilation is performed on the image, which expands the boundaries of the object and tries to fill the holes inside; then, erosion is performed on the dilated image, which removes the protrusions caused by the dilation and pulls the boundaries of the object back to near the original position.

[0045] Step S1c: Select the most significant or largest area from the image after morphological reconstruction and closing operations as the final mask for characterizing the most significant shadow area in the image, and generate a mask library based on it.

[0046] Specifically, after completing the morphological reconstruction and closing operations, the most significant or largest region is selected from the image as the final mask, which can be done by any of the following methods: region growing method, region merging method, threshold-based method, and morphological feature analysis method. The region growing method refers to starting from a seed point and adding adjacent pixels to the region according to certain criteria (such as the similarity of pixel values) until a specific stop condition is met. The region merging method refers to merging adjacent regions into larger regions, usually based on the similarity or connectivity of the regions. The threshold-based method refers to distinguishing the foreground and background by setting a threshold, and then selecting the region exceeding the threshold as the final mask. The morphological feature analysis method refers to analyzing the morphological features of the region, such as area, perimeter, shape, etc., and selecting the most significant feature region as the mask. The final mask library is a collection that stores a series of predefined masks, which are usually used for specific image processing tasks, such as image segmentation, feature extraction, detection, etc. Each mask is a binary image, in which white or non-zero pixels represent the region of interest (ROI), and black or zero pixel areas represent regions of no interest. By applying these masks, local operations can be performed on the image quickly without affecting other parts of the image.

[0047] Preferably, before selecting the most significant or largest region from the image after morphological reconstruction and closing operations, the image is first subjected to edge smoothing processing to reduce noise and discontinuous pixels, making the image look clearer and more natural.

[0048] Exemplarily, the methods of smooth edge processing include but are not limited to: Gaussian filtering, median filtering, bilateral filtering, mean filtering, convex hull detection and filling, image smoothing function method in OpenCV, etc. Gaussian filtering performs weighted averaging of images through Gaussian functions to reduce image noise. In deep learning, the convolution layer in the convolutional neural network can be used to implement Gaussian filtering, and the degree of smoothing can be controlled by adjusting the parameters of the convolution kernel. Value filtering is a nonlinear filtering technique that replaces the value of each pixel in the image with the median of the pixel value in the neighborhood of the point. Bilateral filtering is a filtering method that combines spatial proximity and pixel value similarity, which can maintain edge information while smoothing the image. This method can be implemented in deep learning by constructing a convolution kernel containing two dimensions of spatial distance and pixel value difference. Mean filtering is to achieve smoothing by averaging all pixels in the neighborhood of a certain pixel and then assigning the average value to the pixel. The convex hull detection and filling method uses the convexHull() method to find the convex hull of the contour points, and then uses the fillConvexPoly() function to redraw the area included in the convex hull points, and adds cv::LINE_AA to achieve anti-aliasing effect, thereby smoothing the edges. OpenCV provides a function cvSmooth() that can implement a variety of smoothing methods, such as simple blur, Gaussian blur, median blur, bilateral filtering, etc. This function achieves different smoothing effects by setting different parameters.

[0049] Preferably, after selecting the most significant or largest area from the image after morphological reconstruction and closing operations, the contour of the image is convexed to make it a convex shape, which can improve the recognition of objects and the analysis of the shape characteristics of objects. For example, when processing fundus images, it may be necessary to convex the contours of blood vessels or lesions to facilitate more accurate analysis of their shape and size. Convex processing can also help eliminate irregularities or noise on the contour, making the image look clearer and more regular.

[0050] Exemplarily, the convex processing of the image can be achieved by any of the following methods: OpenCV's convexHull function, morphological image processing, convex hull algorithm, OpenCV library, etc. The OpenCV library provides a function called convexHull, which can calculate the convex hull of a given contour. First, you need to use a function such as findContours to detect the contours in the image, and then use the convexHull function for each contour to obtain its convex hull. The convex hull can be drawn on the original image with a red line to identify the convex shape of the object. Morphological processing is a technique for processing binary images, which includes corrosion and expansion operations. By combining opening and closing operations, the image contour can be smoothed and small holes and breaks inside the object can be filled, which helps to achieve convex processing of the image. The convex hull algorithm is a method for finding the convex hull of connected areas in a binary image. First, you need to find all connected areas, then calculate the center of mass of each area, and find the smallest convex polygon containing the original shape. In Python, you can use the OpenCV library to implement image convexification. First, preprocess the image, such as grayscale conversion and binarization, then use the findContours function to detect contours, and use the convexHull function to calculate the convex hull. Finally, draw the convex hull on the image to intuitively demonstrate the convexification effect.

[0051] Step S2: adding noise to the original image based on the mask library; constructing a noise prediction model based on the original image and the noise image; based on the noise prediction model, training a U-Net model to obtain a diffusion model for restoring the original image from the noise image.

[0052] In the embodiment of the present application, the above step S2 can be divided into the following steps:

[0053] Step S2a: randomly selecting a mask from the mask library and applying it to the original image to generate a corresponding mask image, and adding noise only in the mask area of ​​the mask image.

[0054] Taking fundus image as an example, the process is initiated by selecting a random mask from a pre-established mask library and applying it to the original fundus image, thereby generating a masked image. Noise is added to the masked region of the masked image, ensuring that the noise is applied only to the masked region while keeping the rest of the image unchanged.

[0055] Step S2b: construct a noise prediction model; the noise prediction model is used to describe the mapping relationship between the noise image and the original image.

[0056] Given a noisy image X T , the goal is to reconstruct the image to its original state X0. The diffusion process of forward noise addition is described by the following formula:

[0057] in:

[0058]

[0059] Here, we describe a Markov process where each state is a deterministic transformation of the previous state plus some random noise, and the transformations are given by Scaling, noise by β t Specifically, q(X t |X t-1 ) indicates that given the previous state X t-1 Lower state X t The conditional probability distribution of And the covariance matrix is ​​β t Normal distribution of I (denoted by the symbol N); mean Indicates the current state X t is the previous state X t-1 A scaled version of Covariance matrix β t I, β t is a scalar representing the noise level or uncertainty of the process, and I is the identity matrix, indicating that the covariance matrix is ​​diagonal and the variance of each dimension is β t .

[0060] Step S2c: construct a U-Net model, use the Gaussian process model to define the loss function, and adjust the parameters of the U-Net model through a gradient descent algorithm until the loss function is minimized.

[0061] It should be noted that the U-Net model is a deep learning network, and its core features include: U-shaped structure, encoder, decoder, jump connection, and output layer. The U-shaped structure refers to the U-shaped network structure of U-Net, which includes a contraction path (encoder) and an expansion path (decoder). The contraction path is implemented through a series of convolutional layers and maximum pooling layers. After each pooling, the size of the feature map is halved, and the number of channels is usually doubled to capture a wider range of contextual information; the expansion path enlarges the size of the feature map through upsampling (such as deconvolution or upsampling layers), and introduces features from the encoder part of the corresponding level through jump connections. These features contain rich spatial detail information and help improve segmentation accuracy. In the jump connection, the upsampled features of each layer are combined with the output of the corresponding encoder layer through the concatenation operation, which not only conveys fine spatial information, but also helps alleviate the possible information loss problem during the upsampling process. Finally, the output layer is a feature map of the same size as the input image, and a sigmoid or softmax activation function is usually used to predict the category probability of each pixel.

[0062] In the embodiments of the present application, very clean images are used to train the U-Net model. This is because clean images contain less noise, which can prevent noise from interfering with the network's learning process and causing the model's performance to deteriorate. In medical image analysis, clean images can reduce background noise, allowing the network to more accurately identify and segment target structures.

[0063] In the embodiment of the present application, the unmasked area in the image that does not need to be restored is called X cond , t obeys a uniform distribution and its value range is between the set {1,2,…,T}, and defines γ t =1-β t , And the noise model is defined as follows:

[0064]

[0065] Here, X T is the final noisy image, X0 is the original image, and ∈ is the Gaussian noise with mean 0 and variance the identity matrix I.

[0066] In order to achieve the optimal prediction of the noise vector, the U-Net model is used to learn the noise, and this step is defined as f θ , the learning goal is to minimize the following expectation:

[0067]

[0068] Using the gradient descent strategy, the U-Net model is continuously trained based on the gradient of the loss function until convergence. The loss function is as follows:

[0069]

[0070] Formula (3) defines the learning objective of the U-Net model, which is to minimize the expected value of the difference between the predicted noise and the true noise. Formula (4) specifies the form of the loss function, which aims to calculate the gradient of the loss function. To update the model parameters θ to minimize the loss function. Specifically, Indicates the unmasked area X cond and the noisy image X T The expected value of E (∈,α) represents the expected value of noise ∈ and other parameters such as α. θ (X cond ,X T ,γ) is the output of the U-Net model, indicating that cond and the noisy image X T In the case of , the model predicts the noise, and defines γ represents the proportion of data that is retained. L represents the difference between the predicted noise and the real noise ∈ p The p-th power of the norm, which is used to measure the difference between the noise predicted by the model and the actual noise.

[0071] Step S3: Use the trained diffusion model to perform model inference to reconstruct the shadowed image into a clean image without loss.

[0072] Specifically, step S3 uses the Markov process to perform reverse reasoning for image denoising. Given the current state, the future state of the process is conditionally independent of the past state. In the context of image denoising, from the noisy image X T The transformation to a clean original image X0 can be modeled as a series of state transitions, where each state only depends on the previous state and some conditional inputs.

[0073] The probabilistic model of the inverse process is given by:

[0074]

[0075] Among them, p θ (X t-1 |X t ,c) is given the current noise image X t Under condition c, reverse step X t-1 The probability distribution of is modeled as a Gaussian distribution as follows:

[0076]

[0077] Among them, μ θ (X t ,c,t) is the mean of the Gaussian distribution, is the covariance matrix.

[0078] The mean μ of the Gaussian distribution θ (X t ,c,t) is given by:

[0079]

[0080] Among them, μ θ (X cond ,X T ,γ T ) represents the mean value predicted by the model at time step T, α t represents the attenuation factor used to control noise, f θ (X cond ,X T ,Y t) is the predicted value output by the U-Net model.

[0081] Final ImageX T-1 It is gradually obtained by iterative updating as follows:

[0082]

[0083] Among them, ∈ t is a random noise term that is added to the updated image to simulate the uncertainty in the inverse process.

[0084] For ease of understanding, combined Figure 3 The flowchart of fundus image processing fully demonstrates the model training process and model reasoning process of the embodiment of the present application:

[0085] Step 1: Randomly select a mask from the mask library and apply it to the original image to generate the corresponding mask image.

[0086] Step 2: Add random noise to the masked area of ​​the mask image.

[0087] Step 3: Model training, from the final state noisy image X T Iterate to get the image X of the previous state T-1 , and then according to the image X T-1 Iterate the image X of the previous state T-2 , and so on, until the image X1 is obtained through iteration, and finally the original image X0 is reconstructed.

[0088] For ease of understanding, combined Figure 4 The flowchart of fundus image processing in the figure fully demonstrates the technical solution of the embodiment of the present application: This figure describes the image restoration process of the fundus image quality enhancement network of the ClarityDiffuseNet model (the model of the present application is named as such in this embodiment):

[0089] Step 1: Convert the original image with lower quality to CMYK format.

[0090] Step 2: Separate the black plane from the CMYK format image.

[0091] Step 3: Perform quantitative analysis on the black plane in the CMYK format, and establish two dynamic thresholds for distinguishing different areas in the image based on the quantitative analysis results.

[0092] Step 4: Smooth the image to reduce noise and discontinuous pixels, making the image look clearer and more natural.

[0093] Step 5: Select the most salient or largest region from the image after morphological reconstruction and closing operations as the final mask to characterize the most salient shadow region in the image.

[0094] Step 6: Randomly select a mask from the mask library and apply it to the original image to generate a corresponding mask image, and add noise only in the mask area of ​​the mask image.

[0095] Step 7: Use the model of this application to form the final reconstructed image.

[0096] Since the model training process, reasoning process and the entire implementation process have been described in detail in the above embodiments, they will not be repeated here.

[0097] It should also be noted that, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects.

[0098] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" represent examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0099] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0100] Figure 5 is a schematic block diagram of a fundus color photo restoration system based on a diffusion model provided in an embodiment of the present application. Figure 5 As shown, the system includes: a mask library construction module 51, a model training module 52, and a model reasoning module 53.

[0101] The mask library construction module 51 is used to convert a number of color images into CMYK format, quantize the black plane to establish a dynamic threshold, apply a binary mask to perform morphological operations, select the largest area in each image as a mask to identify the most prominent shadow area, and construct a mask library based on this. The model training module 52 is used to add noise to the original image based on the mask library; construct a Gaussian process model based on the original image and the noisy image; based on the Gaussian process model, train the U-Net model to obtain a diffusion model for recovering the original image from the noisy image. The model inference module 53 is used to perform model inference using the trained diffusion model to reconstruct the shadowed image into a clean image without loss.

[0102] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here. It should also be understood that the division of modules in the embodiment of the present application is schematic, which is only a logical function division, and there may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application can be integrated into a processor, or it can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0103] Figure 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. Figure 6 As shown in FIG. 6 , the computer device includes: at least one processor 601, a memory 602, at least one network interface 603, and a user interface 605. The various components in the device are coupled together via a bus system 604. It can be understood that the bus system 604 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 604 is not described in detail. Figure 6 In the specification, various buses are labeled as bus systems.

[0104] The user interface 605 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0105] It is understood that the memory 602 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0106] The memory 602 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 600. Examples of these data include: any executable program used to operate on the electronic terminal 600, such as an operating system 6021 and an application 6022; the operating system 6021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 6022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The method for restoring fundus color photography based on a diffusion model provided in the embodiment of the present invention can be included in the application 6022.

[0107] The method disclosed in the above embodiment of the present invention can be applied to the processor 601, or implemented by the processor 601. The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 601 or the instruction in the form of software. The above processor 601 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 601 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 601 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0108] In an exemplary embodiment, the electronic terminal 600 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0109] According to the method provided in the embodiment of the present application, the present application also provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, the computer executes Figures 1 to 4 A method for restoring fundus color photography based on a diffusion model in any of the embodiments shown.

[0110] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code. When the program code runs on a computer, the computer executes the above method.

[0111] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components can reside in a process and / or an execution thread, and a component can be located on a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures stored thereon. Components can, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0112] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0114] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0115] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0117] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When loading and executing computer program instructions (programs) on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs), etc.).

[0118] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0119] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0120] In summary, the present application provides a method, system, device, medium and program product for restoring fundus color photographs based on a diffusion model. The technical solution of the present application not only improves the accuracy and efficiency of image processing, but also enhances the generalization ability and robustness of the model. For existing algorithms (such as generative adversarial networks or autoencoders) that have limited utility when images suffer from significant detail loss, the present application can well restore the details and accuracy of the image. Therefore, the present application effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0121] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for restoring fundus color photos based on a diffusion model, characterized in that: include: Convert several color images to CMYK format, quantize the black plane to establish a dynamic threshold, apply binary masks to perform morphological operations, select the largest area in each image as a mask to identify the most significant shadow area, and build a mask library based on this; Adding noise to the original image based on the mask library; building a noise prediction model based on the original image and the noise image; Based on the noise prediction model, the U-Net model is trained to obtain a diffusion model for recovering the original image from the noisy image; the trained diffusion model is used for model inference to reconstruct the shadowed image into a clean image without loss.

2. The method for restoring fundus color photos based on a diffusion model according to claim 1, characterized in that: The method converts a plurality of color images into CMYK format, quantizes the black plane to establish a dynamic threshold, applies a binary mask to perform morphological operations, selects the largest area in each image as a mask to identify the most significant shadow area, and constructs a mask library based on the selected area, which includes: Convert several color images into CMYK format, perform quantitative analysis on the black plane in the CMYK format, and establish two dynamic thresholds for distinguishing different areas in the image based on the quantitative analysis results, so as to identify and separate the parts of the area of ​​interest; Based on two dynamic thresholds, a binary mask is applied to perform morphological reconstruction and closing operations on the image; The most significant or largest region is selected from the image after morphological reconstruction and closing operations as a final mask for characterizing the most significant shadow region in the image, thereby generating a mask library.

3. The method for restoring fundus color photos based on a diffusion model according to claim 2, characterized in that: The method is based on two dynamic thresholds and applies a binary mask to perform morphological reconstruction and closing operations on the image, which includes: applying a binary mask to the image using the dynamic threshold to convert the image into a binary image; performing morphological reconstruction and closing operations on the binary image, wherein the morphological reconstruction is used to restore the destroyed structure, and the closing operation is used to eliminate small holes and connect disconnected parts.

4. The method for restoring fundus color photos based on a diffusion model according to claim 1, characterized in that: Adding noise to the original image based on the mask library; building a noise prediction model based on the original image and the noise image; Based on the noise prediction model, a U-Net model is trained to obtain a diffusion model for recovering the original image from the noisy image, which includes: Randomly selecting a mask from the mask library and applying it to the original image to generate a corresponding mask image, and adding noise only in the masked area of ​​the mask image; Constructing a noise prediction model; the noise prediction model is used to describe the mapping relationship between the noise image and the original image; A U-Net model is constructed, and the noise prediction process model is used to define the loss function. The parameters of the U-Net model are adjusted by the gradient descent algorithm until the loss function is minimized.

5. The method for restoring fundus color photos based on a diffusion model according to claim 4, characterized in that: The construction process of the noise prediction model includes: Given a noisy image X T , the goal is to reconstruct the image to its original state X0, and the k-diffusion process of adding noise forward is: Among them, q(X t |X t-1 ) indicates that given the previous state X t-1 Lower state X t The conditional probability distribution of And the covariance matrix is ​​β t Normal distribution of I, represented by the symbol N; mean Indicates the current state X t is the previous state X t-1 A scaled version of Covariance matrix β t I, β t is a scalar representing the noise level or uncertainty of the process, and I is the identity matrix, indicating that the covariance matrix is ​​diagonal and the variance of each dimension is β t .

6. The method for restoring fundus color photos based on a diffusion model according to claim 4, characterized in that: The learning objectives of the U-Net model are: The loss function of the U-Net model is: in, Indicates the unmasked area X cond and the noisy image X T The expected value of E (∈，α) represents the expected value of noise ∈ and parameter α; f θ (X cond , X T ,γ) is the output of the U-Net model, indicating that in a given unmasked area X cond and the noisy image X T In the case of , the model predicts the noise, γ represents the proportion of data retained; represents the difference between the predicted noise and the real noise ∈ p The p-th power of the norm, which is used to measure the difference between the noise predicted by the model and the actual noise.

7. A fundus color photo restoration system based on a diffusion model, characterized in that: include: A mask library construction module is used to convert several color images into CMYK format, quantize the black plane to establish a dynamic threshold, apply binary masks to perform morphological operations, select the largest area in each image as a mask to identify the most significant shadow area, and construct a mask library based on this; A model training module, used for adding noise to the original image based on the mask library; and building a noise prediction model based on the original image and the noise image; Based on the noise prediction model, a U-Net model is trained to obtain a diffusion model for recovering an original image from a noisy image; The model inference module is used to perform model inference using the trained diffusion model to reconstruct the shadowed image into a clean image without loss.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for restoring fundus color photography based on a diffusion model as described in any one of claims 1 to 6 is implemented.

9. A computer program product, characterized in that The computer program product includes computer program codes, and when the computer program codes are executed on a computer, the computer is enabled to implement the method for restoring fundus color photography based on a diffusion model as claimed in any one of claims 1 to 6.

10. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method for restoring fundus color photography based on a diffusion model as described in any one of claims 1 to 6.

Images