Training method and reconstruction method of glaucoma fundus image super-resolution generation model

Through the super-resolution generation model of glaucoma fundus image, the generative adversarial network and U-Net network are used, combined with denoising and segmentation technology, the problem of OCT technology being difficult to capture subtle deformation of the sieve hole is solved, achieving high-quality image reconstruction, reducing the Early diagnosis is difficult and improves the availability of 3D reconstruction.

CN118967442APending Publication Date: 2024-11-15HARBIN INST OF TECH
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410997252.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing OCT technologies are difficult to accurately capture the subtle deformation and complex structure of glaucoma sieve holes, resulting in blur or distortion of images, increasing the difficulty and uncertainty of early diagnosis.

Method used

A super-resolution generation model for glaucoma fundus images is adopted, based on the Generative Adversarial Network (GAN) and U-Net network, and a loss function guide generator to generate high-quality super-resolution images, combining Markov chain denoising and Curvelet transformation Segmentation technology optimizes image processing to improve diagnostic reliability.

Benefits of technology

It significantly improves the quality and clarity of OCT images, provides images that meet medical evaluation indicators, reduces the difficulty of diagnosis in early glaucoma patients, and improves the availability of three-dimensional reconstruction of screen morphology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967442A_ABST
    Figure CN118967442A_ABST
Patent Text Reader

Abstract

The invention discloses a training method and a reconstruction method of a glaucoma fundus image super-resolution generation model, and belongs to the technical field of medical image data processing. In order to solve the problem that when super-resolution reconstruction is carried out on a glaucoma fundus image through an existing generative model, reconstruction cannot be effectively carried out according to the characteristics of the glaucoma fundus image, the reconstruction generative model is a generator of a generative adversarial network, and the high-resolution glaucoma fundus image serves as a reference image; a low-resolution image obtained by down-sampling the fundus image is used as the input of a generator, and the generator can generate a super-resolution reconstruction image of the fundus image of glaucoma through the overall training of the generative adversarial network, and in the training process of the generative adversarial network, the super-resolution reconstruction image of the fundus image of glaucoma is obtained. And calculating image quality evaluation loss and edge similarity loss by using the cross section area of the nth nerve fiber layer of the reference image and the reconstructed image and the binary edge image of the sieve plate region as total loss to realize model training.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image data processing, and in particular relates to a training method and a reconstruction method for a fundus image super-resolution generation model. Background Art

[0002] Changes in the ethmoid holes of glaucoma patients are an important pathological phenomenon. The ethmoid holes are tiny holes in the scleral sieve plate that allow the optic nerve fibers to pass through. In glaucoma patients, as the intraocular pressure continues to rise, the sieve plate is subjected to increasing pressure, and the sieve hole structure undergoes changes such as deformation and distortion. These changes may squeeze the optic nerve fibers passing through the sieve holes, causing damage to the optic nerve fibers. Therefore, timely detection and control of intraocular pressure and reduction of pressure on the sieve plate are the key to preventing further deformation of the sieve plate and sieve hole structure and optic nerve damage.

[0003] Optical coherence tomography, or OCT, has obvious advantages in the current early diagnosis of glaucoma, but it also has some limitations. First, due to the complexity and subtlety of the scleral sieve plate and sieve hole structure, and the small deformation of the sieve holes in early glaucoma, the existing OCT technology is difficult to accurately capture its detailed structure and subtle deformation, resulting in blurred or distorted image structure and difficulty in analysis. Secondly, factors such as eye movement, interference from the tear layer, and differences in eye structure between different patients may affect the quality of OCT image acquisition, thereby increasing the difficulty and uncertainty of doctors' diagnosis of the disease.

[0004] Image super-resolution reconstruction, Image super-resolution, refers to the process of upgrading low-resolution images to high-resolution images by using computer algorithms. This technology is very important in the fields of computer vision and image processing. The goal of image super-resolution reconstruction is to make low-resolution images visually closer to high-resolution images by increasing the details and clarity in the image. Commonly used methods include interpolation methods, edge-based methods, statistics-based methods, and deep learning-based methods. Among them, the deep learning-based method uses deep neural networks to learn the mapping from low-resolution images to high-resolution images, has a good reconstruction effect, and can automatically learn features. Summary of the invention

[0005] The present invention aims to solve the problem that when using the existing generation model to perform super-resolution reconstruction of glaucoma fundus images, the reconstruction cannot be effectively performed according to the characteristics of the glaucoma fundus images, so that the reconstructed images cannot be well used as images that meet medical evaluation indicators.

[0006] A training method for a super-resolution reconstruction generative model of a glaucoma fundus image, wherein the super-resolution reconstruction generative model of a glaucoma fundus image is a generator of a generative adversarial network, a high-resolution glaucoma fundus image is used as a reference image, and a low-resolution image obtained by downsampling the high-resolution glaucoma fundus image is used as an input of the generator, and the generator is enabled to generate a super-resolution reconstruction image of the glaucoma fundus image through overall training of the generative adversarial network. During the training process of the generative adversarial network, a total loss function is obtained by using a loss function F, and the training of the generator is guided based on the total loss function;

[0007] The loss function F is as follows:

[0008] F=w1DCA_NFL(I ref ,I rec )+w2LC_ESM(I ref ,I rec )

[0009]

[0010]

[0011] Among them, w1 and w2 are adjustment weights, DCA_NFL(I ref ,I rec ) represents the image quality assessment loss, LC_ESM(I ref ,I rec ) represents edge similarity loss; A rec,n A ref,n represents the cross-sectional area of ​​the nerve fiber layer of the reference image and the reconstructed image, respectively, and N is the number of nerve fiber layers obtained by segmentation; E ref E rec are the binary edge maps of the lamina cribrosa region of the reference image and the reconstructed image respectively; |E ref ∩E rec | is the number of pixels where the binary edge of the lamina cribrosa region in the reference image overlaps with that in the reconstructed image; |E ref +E rec | is the sum of the number of binary edge pixels in the lamina cribrosa region in both the reference image and the reconstructed image, and is the total number of edge pixels in the two images.

[0012] Furthermore, the generative adversarial network is a SRGAN network.

[0013] Furthermore, during the training process of the generative adversarial network, the output of the generator in the generative adversarial network is guided by generating a mask to determine the nerve fiber layer area and the cribriform plate area.

[0014] Furthermore, the mask generation method is implemented using a U-Net network.

[0015] Furthermore, in the training process of the generative adversarial network, the high-resolution glaucoma fundus image is a high-resolution glaucoma fundus image obtained by denoising the original glaucoma fundus image and then segmenting the OCT image based on Curvelet transform and watershed. In the OCT image segmentation process based on Curvelet transform and watershed, the glaucoma fundus image is firstly subjected to Curvelet transform, and then segmented using the OTSU algorithm.

[0016] Furthermore, the images segmented by the OTSU algorithm are subjected to morphological processing to finally obtain high-resolution glaucoma fundus images for training the generative adversarial network.

[0017] Furthermore, the denoising process of the original glaucoma fundus image is implemented based on the Markov chain. In the denoising process of the Markov chain, for the image q(y t-1 ), the image before denoising in the first step is the original glaucoma fundus image to be denoised, and the corresponding noise ε is first predicted using UNet T , and then use the following formula to denoise the current step;

[0018]

[0019] Among them, β t is the denoising coefficient;

[0020] In the denoising process of the Markov chain, the denoising process of the current step is cycled to finally obtain a denoised high-resolution glaucoma fundus image.

[0021] Furthermore, in the process of segmentation using the OTSU algorithm, the calculation formula for the inter-class variance is as follows:

[0022] γ 2 =ω1ω2(ρ1-ρ2) 2 W

[0023] Among them, ω1 and ω2 are the proportion of pixels below the threshold τ and the proportion of pixels above the threshold τ respectively; ρ1 and ρ2 are the average grayscale values ​​of pixels below the threshold τ and the average grayscale values ​​of pixels above the threshold τ respectively; W is the weighting factor.

[0024] A method for super-resolution reconstruction of glaucoma fundus images uses a trained generator obtained by a training method for a glaucoma fundus image super-resolution reconstruction generation model to reconstruct a low-resolution glaucoma fundus image to be reconstructed, thereby obtaining a reconstructed super-resolution glaucoma fundus image.

[0025] Furthermore, before the low-resolution glaucoma fundus image to be reconstructed is input into the generator, it is necessary to denoise the low-resolution glaucoma fundus image to be reconstructed. The process of denoising the low-resolution glaucoma fundus image to be reconstructed is based on the Markov chain. In the denoising process of the Markov chain, for the image q(y t-1 ), the image before denoising in the first step is the low-resolution glaucoma fundus image to be denoised, and the corresponding noise ε is first predicted using UNet T , and then use the following formula to denoise the current step;

[0026]

[0027] Among them, β t is the denoising coefficient;

[0028] In the denoising process of the Markov chain, the denoising process of the current step is cycled to finally obtain a denoised low-resolution glaucoma fundus image.

[0029] Compared with the existing technologies and methods, the biggest advantages of the present invention are:

[0030] 1. The present invention strengthens the characteristics of OCT images by preprocessing them, and uses a neural network that introduces medical indicators as image quality assessment based on SRGAN to realize the training of the reconstruction network model. The reconstruction network model reconstructs the image with super-resolution, which can greatly improve the quality and simplify the information, and obtain images that meet the medical evaluation indicators, providing better input for the network for early auxiliary diagnosis of subsequent glaucoma diseases and the three-dimensional reconstruction of the cribriform plate morphology.

[0031] 2. The present invention introduces a medically significant image quality assessment method into the current super-resolution reconstruction technology based on adversarial production networks. Through optimization training, an enhanced network that can be used for OCT image processing is obtained, which can achieve super-resolution reconstruction of OCT images of glaucoma patients, reduce the difficulty of diagnosis for early glaucoma patients, and improve the usability of OCT images for three-dimensional reconstruction of sieve plate morphology. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is an imaging effect diagram of the thickness of the retinal nerve fiber layer and the morphology of the lamina cribrosa. Figure 2 Flowchart of the super-resolution reconstruction method for glaucoma fundus images. Figure 3 This is the network structure diagram of the Generative Adversarial Network (SRGAN). DETAILED DESCRIPTION The present invention is based on the current residual learning and adversarial generative network super-resolution reconstruction technology, and introduces the medically significant image quality assessment method and U-Net network generation mask. Applying the trained enhancement network to the processing of OCT images can greatly improve the image quality and simplify the information. SRGAN+U-NET (Residual adversarial generative network and U-Net network generation mask) The present invention combines medical knowledge with computer science and technology, uses computer vision to analyze and process medical imaging data, and provides strong support for the early diagnosis and stage evaluation of degenerative diseases.

[0033] Specific implementation method 1: Combination Figure 2 To explain this embodiment,

[0034] The content is a super-resolution reconstruction method for glaucoma fundus images based on SRGAN+U-NET, which includes the following steps:

[0035] S100, data set analysis and processing:

[0036] S101, Image denoising based on diffusion model:

[0037] The diffusion model gradually transforms random noise into target data distribution by simulating the random diffusion process, thereby generating new data samples. In OCT image denoising, the diffusion model can use partial differential equations to describe the diffusion process of noise in the image based on the difference between the grayscale values ​​of pixels in the image to achieve the purpose of denoising. This method can effectively remove noise from OCT images, improve image clarity and contrast, and thus improve image interpretation and analysis. The specific steps are as follows:

[0038] Diffusion process: gradually mix the noise-free OCT image with Gaussian noise to generate a noisy OCT image.

[0039] This process can be regarded as a Markov chain process, where each step changes the pixel value to a value closer to the Gaussian distribution. As the steps proceed, the noise gradually increases and the image quality gradually decreases. Based on the difference between the grayscale values ​​of pixels in the image, a partial differential equation is used to describe the diffusion process of noise in the image.

[0040] Markov chain: It is a "memoryless transition" from one state in the state space to another, so that the state at a certain moment is only related to the state at the previous moment. Markov chain can be expressed as:

[0041] P(x t+1 |…, x t-2 , x t-1 , x t )=P(x t+1 |x t )

[0042] Among them, x t is a certain moment in the transformation, P(x t ) corresponds to x t The state at the moment.

[0043] OCT is based on the principle of low-coherence interference, and speckle noise will inevitably appear in the image. However, due to the lack of a large number of paired high-noise images and low-noise images, it is necessary to add noise to the low-noise image. For this purpose, we define a forward Markov diffusion process:

[0044]

[0045] Among them, y t represents the OCT image after adding noise in the tth step, y0 represents the initial noise-free OCT image, and y T represents the final OCT image after T-step noise addition, q(y t |y t-1 ) represents the state function corresponding to the OCT image obtained by adding noise to the OCT image at step t-1, that is, the state function of the image after each noise addition;

[0046] Forward process (noise addition process): Each noisy process gradually adds Gaussian noise to the high-resolution image through T iterations. The probability density function (PDF) of the Gaussian distribution can be expressed as:

[0047]

[0048] The mean μ of the Gaussian noise is usually fixed to 0, and the standard deviation σ needs to be determined according to the required noise level.

[0049] In the process of adding Gaussian noise to the image, a noise array is generated according to the probability density function of the above Gaussian distribution and the image size, and then the noise array is added to the image of the current state to obtain the next state image, and the noise addition process is gradually completed:

[0050]

[0051] Among them, q(y t ) is y t The final OCT image at time q(y t-1 ) is the final OCT image at the previous moment, ε t is the noise to be added obtained from Gaussian distribution; α t is the diffusion coefficient, which is not determined by the training process and is a monotonically increasing function in time, adding less noise in the early steps and more noise in the later steps;

[0052] Reverse operation (denoising process): After obtaining the noisy image, gradually restore the noisy image to the noise-free image.

[0053] This process is also a Markov chain process, and each step adjusts the pixel value to a value closer to the original noise-free image. As the steps proceed, the noise is gradually removed, and the denoised image is finally obtained.

[0054] In the reverse process, the model also follows a Markov chain, but in reverse. This means that in the reverse step, the model only needs to know the state and noise at time t-1 to predict the state at time t.

[0055] For the forward process, randomly add Gaussian normal distribution OCT images q(y t ), and the corresponding noise ε is predicted by U-Net T , similar to the forward process requires q(y y-1 ) to q(y t ) process to add noise ε t , denoising also requires q(y t ) to remove the noise ε T Get q(y t-1 ), and so on, finally a high-quality x0 image is obtained.

[0056]

[0057] Among them, β t is the denoising coefficient;

[0058] Parameter selection: Choose an appropriate diffusion coefficient α T and denoising coefficient β t It is the key, the choice of parameters will affect the denoising effect and image quality.

[0059] The diffusion coefficient and denoising coefficient are the degree of adding and removing noise at each step in the training process. These two coefficients are pre-set, and the determination of the coefficients usually requires experimental verification. By testing different coefficient configurations on the validation set, the setting that produces the best performance is selected based on the verification results. If the diffusion coefficient is set too large, over-diffusion will occur, image features will be seriously lost, and it will be difficult to restore a high-quality image; if the diffusion coefficient is too small, under-diffusion will occur, and it will be too similar to the original data. If the denoising coefficient is set too small, more reverse steps may be required to generate data, resulting in a slower generation process. At this time, the denoising coefficient needs to be increased; if the denoising coefficient is too large, it will make it difficult for the network to converge to a high-quality solution. These phenomena will be reflected in the mean square error MSE, the structural similarity coefficient SSIM, and the peak signal-to-noise ratio PSNR.

[0060] OCT technology is a high-resolution, non-contact, tomographic imaging device that can be used to examine the posterior segment of the eye, mainly the thickness of the retinal nerve fiber layer and the lamina cribrosa. Figure 1 Noise will appear during the imaging process. For noisy OCT images, the trained network is used to remove noise.

[0061] S102, OCT image segmentation based on Curvelet transform and OTSU:

[0062] In order to obtain clearer retinal nerve fiber layer structure and lamina cribrosa morphology and richer details, the denoised OCT images were processed using image segmentation algorithm.

[0063] For the denoised OCT image, first use Curvelet transform to perform multi-scale analysis on the OCT image to extract the curve and edge features in the image. Then combine the maximum inter-class variance method, that is, the OTSU improved algorithm, to determine the appropriate segmentation threshold according to the grayscale characteristics of the image, and segment the image according to the threshold to obtain an OCT image with obvious features after segmentation. This algorithm can achieve accurate image segmentation while retaining image details. The specific steps are as follows:

[0064] (1) Curvelet transform of image:

[0065] According to the characteristics of the coefficients at different transformation scales, the transformation coefficients are processed, and then the image is inversely transformed to obtain an enhanced image.

[0066] Since Curvelet transform has multi-scale, anisotropic characteristics and can better approximate the edge of the image, the edge information of the image can be enhanced by modifying its transform coefficients.

[0067] (2) For images enhanced by Curvelet transform, the improved OTSU algorithm is used for segmentation:

[0068] The OTSU method is a global threshold method. It traverses the pixel level of the image to find the threshold τ that maximizes the inter-class variance, which is the required segmentation threshold. The calculation formula for the inter-class variance is as follows:

[0069] γ 2 =ω1ω2(ρ1-ρ2) 2 W

[0070] Among them, ω1 and ω2 are the proportion of pixels below the threshold τ and the proportion of pixels above the threshold τ, respectively; ρ1 and ρ2 are the average grayscale values ​​of pixels below the threshold τ and the average grayscale values ​​of pixels above the threshold τ, respectively. W is a weighting factor, which is usually not used in the standard OTSU method. For fundus OCT images, there is a problem of blurred edges between the foreground and the background. For this, the gradient weight factor is used to improve the segmentation effect. First, the gradient of the image is calculated. The gradient reflects the intensity and direction of the grayscale change of the pixels in the image. The gradient is used as a weight factor and multiplied with the grayscale probability distribution of the pixels at the corresponding position. This can increase the importance of areas with large gradient changes, and can make the OTSU method pay more attention to the edges during segmentation, thereby improving the segmentation performance.

[0071] The improved OTSU algorithm can select a suitable threshold based on the image grayscale histogram to distinguish between two categories to achieve the purpose of image segmentation. In addition, the optimal threshold of the traditional OTSU algorithm will have a "threshold offset" phenomenon to varying degrees. The improved OTSU algorithm adds gradient weights, which basically eliminates the interference and image noise caused by low contrast, and the contours of the retinal nerve fiber layer are clearer and the details are richer, and the effect is significantly improved.

[0072] (3) Morphological image post-processing:

[0073] The segmentation results of the improved OTSU algorithm have extracted the foreground including the retinal nerve fiber layer and the lamina cribrosa. However, since some areas in the retinal nerve fiber layer and the lamina cribrosa have grayscale values ​​similar to the background, there are still some isolated points and holes. The image is processed using the connected regions in morphology, and denoising is performed by judging the number of pixels in the connected regions, that is, the area of ​​the connected regions.

[0074] Isolated points and holes are not connected to other areas. You can count the number of pixels of these isolated points and holes, set a threshold, and if the number of pixels is lower than the threshold, it will be considered as noise and eliminated. The setting of the threshold needs to be determined according to the actual situation. Under different resolutions and field of view, there should be different judgment criteria for whether isolated points and holes are valid information, and it should be set according to the actual image.

[0075] S200, super-resolution reconstruction based on SRGAN+U-NET:

[0076] SRGAN is a generative adversarial network (GAN) that introduces residual learning. The input during training is a high-resolution image as the original image. During training, the original image is downsampled to obtain a low-resolution image. The low-resolution image is reconstructed based on the generative adversarial network. The generator is trained to generate a reconstructed image that is as similar to the original image as possible, and the discriminator is trained to distinguish between the original image and the reconstructed image. The training of the two promotes each other and improves the performance of network reconstruction. Based on the SRGAN network, the present invention defines a new image evaluation function to guide network training, adds a U-Net network to generate a mask for local image evaluation, and at the same time improves the effect of the SRGAN network in the OCT image super-resolution reconstruction task;

[0077] In the process of training the generator, an objective function with medical significance is introduced, and attention is paid to whether the indicators with medical significance are consistent between the reconstructed image and the original image, so as to determine whether the network has achieved the effect of super-resolution reconstruction.

[0078] Traditional image quality assessment indicators, such as PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index), have the disadvantages of insufficient subjective perception, insufficient structural complexity, lack of medical value, sensitivity to noise and artifacts, and susceptibility to misleading by information-free areas when used for super-resolution reconstruction of fundus OCT images. Therefore, the present invention also proposes an image quality assessment method that introduces medical indicators, including:

[0079] 1. Pre-trained U-Net network generates masks for reconstruction and image quality assessment

[0080] The U-Net network is trained using OCT images and masks with key areas (including clear nerve fiber layer and lamina cribrosa) marked, so that the trained network can receive OCT images as input and output masks that can shield invalid information areas and highlight medical features. The generated masks are used to guide the network to focus on key areas such as the nerve fiber layer and lamina cribrosa during super-resolution reconstruction and image reconstruction quality assessment. This part is explained in detail in the subsequent mask generation network (U-Net).

[0081] 2. Introduce the image quality assessment method of medical indicators to evaluate the reconstructed super-resolution images and guide the training process;

[0082] Nerve fiber layer thickness and lamina cribrosa morphology are of great significance for the clinical diagnosis and medical research of glaucoma. Nerve fiber layer thickness is an important indicator for the clinical diagnosis of glaucoma, and three-dimensional reconstruction based on the two-dimensional morphology of the lamina cribrosa is of great significance for studying the pathogenesis of glaucoma. Figure 1 As shown, the nerve fiber layer appears as a layered structure in the OCT image, while the lamina cribrosa appears as a boundary with changing grayscale values.

[0083] The objective function of the super-resolution algorithm is usually to maximize a certain image quality index, as well as the perceptual loss used in SRGAN, pursuing high resolution perceived by the human eye, but these indicators often cannot fully meet the medical requirements for image reconstruction quality, and may result in the loss of detail information commonly used in the clinical diagnosis of glaucoma. Therefore, the present invention proposes an image quality assessment method that introduces medical indicators as an objective function, which is weighted and combined with the existing objective function to guide the training of the neural network. The method includes the following two evaluation indicators, which are specifically defined as follows:

[0084] DCA-NFL-Differences in cross-sectional area of ​​nerve fiber layer:

[0085] This evaluation index is based on the reference image I ref and reconstructed image I rec The measurement of the cross-sectional area of ​​nerve fibers in the nerve fiber layer region. In the process of evaluating the quality of the reconstructed image, the previously generated mask is used to guide the generative adversarial network to detect the cross-sectional area of ​​nerve fibers in each nerve fiber layer in the nerve fiber layer region. The function is defined as:

[0086]

[0087] Among them, A rec,n , A ref,n represent the cross-sectional areas of the nerve fiber layers of the reference image and the reconstructed image, respectively, and N is the number of nerve fiber layers obtained by segmentation. In this embodiment, N is 8.

[0088] This function provides a quantitative comparison to assess how similar the reconstructed image is to the reference image in terms of nerve fiber layer thickness. Normalization is done to make the evaluation independent of image size. The result of this function should be close to 1.

[0089] Lamina Cribrosa-ESM–Lamina Cribrosa Edge Similarity Measure, ESM:

[0090] This evaluation index is based on the reference image I ref and reconstructed image I rec The cribriform plate edge similarity measure reflects the cribriform plate morphology and edge information. In the process of evaluating the quality of the reconstructed image, the previously generated mask is used to guide the network to perform edge similarity detection only on the cribriform plate area. When detecting edge similarity, the Canny operator is first used to detect the edge, the obtained edge is binarized, and the similarity measurement function is defined

[0091]

[0092] Among them, E ref 、E rec are the binary edge maps of the lamina cribrosa region of the reference image and the reconstructed image, respectively. ref ∩E rec | is the number of pixels where the binary edges of the lamina cribrosa region overlap in the two images. |E ref +E rec | is the sum of the number of binary edge pixels in the lamina cribrosa region in the two images, which is the total number of edge pixels in the two images. When the super-resolution reconstruction network has high quality for the lamina cribrosa morphology reconstruction, the result of this function should be close to 1.

[0093] The reconstructed image should have a good reconstruction effect on the thickness of the nerve fiber layer and the lamina cribrosa in the image, and the requirements of the two on the network are similar (i.e., good edge reconstruction effect). Therefore, the two functions defined above can be jointly optimized, that is, they are combined into a joint image quality assessment function and then optimized. The present invention adopts a weighted summation method for the function. The following is the combined function:

[0094] F=w1DCA_NFL(I ref ,I rec )+w2LC_ESM(I ref ,I rec )

[0095] Among them, w1 and w2 correspond to the weights of function 1 and function 2, which are a set of hyperparameters that affect the training effect of the network. For different training objectives and reconstructed image application scenarios, w1 and w2 have different values; for example, if the reconstructed image is used for glaucoma diagnosis, more accurate nerve fiber layer thickness information is required, and function 1 should have a larger weight (w1 is larger); if the reconstructed image is used for three-dimensional reconstruction of the lamina cribrosa, a good reconstruction effect of the lamina cribrosa morphology is required, and function 2 should have a larger weight (w2 is larger).

[0096] In order to optimize the training process of the SRGAN super-resolution generation network, the newly defined objective function combination function F needs to be used in combination with the existing objective function F′. In fact, F is combined with the existing network model loss function to complete the model training, so the objective function F′ can use the existing loss. The existing objective function F ′ Including but not limited to adversarial loss, pixel loss, etc. Since perceptual loss may introduce distortion and afterimages that affect the super-resolution generation of OCT images, perceptual loss is generally not introduced. ′The weighted sum effectively jointly guides the training process.

[0097] In order to better illustrate the loss function of the present invention, the following overall objective function F total The weight parameters will be adjusted according to the actual training results to achieve the best generated image quality. The overall objective function F total The expression is as follows:

[0098] F total =ωF+ω′F′

[0099] Among them, ω and ω′ are weight parameters that need to be adjusted according to the training effect. The existing objective function F′ is expressed as:

[0100] F′=L GAN +λL pixel

[0101] Among them, L GAN and L pixel are adversarial loss and pixel loss, respectively, and λ is the weight parameter adjusted according to the actual training effect. The above weighted combination method helps to balance the contribution of different loss terms during training, thereby generating high-quality super-resolution images.

[0102] 3. Optimizer selection:

[0103] The Adam optimizer, which is widely used in image generation and super-resolution tasks based on the GAN network, is selected.

[0104] Medical indicators such as nerve fiber layer thickness and lamina cribrosa curvature, which are important in glaucoma diagnosis, are introduced into the reconstructed image quality evaluation of the generative adversarial network, making the generated images have practical utility in medicine. These preprocessing steps help improve the quality of the image, highlight the characteristics of the disease, and simplify the information, providing better input for subsequent network processing.

[0105] After training, a trained SRGAN network is obtained. For the low-resolution glaucoma fundus image to be reconstructed, the generator of the trained SRGAN network is used to generate a super-resolution image of the glaucoma fundus image, that is, super-resolution image reconstruction of the glaucoma fundus image. Before the generator reconstructs the image, the low-resolution glaucoma fundus image to be reconstructed can also be denoised (the denoising process is implemented based on a Markov chain), or / and OCT image segmentation processing based on Curvelet transform and watershed, or / and morphological processing.

[0106] The SRGAN+U-NET network used in this paper is an advanced image super-resolution reconstruction method that combines residual learning, generative adversarial networks and U-Net networks. It recovers high-resolution detail information from low-resolution images through a deep learning model. This method combines multiple technical advantages to achieve high-quality and efficient super-resolution reconstruction.

[0107] A. Residual learning plays a key role in this method.

[0108] The network structure of the Generative Adversarial Network (SRGAN) is as follows Figure 3 As shown in the figure, by introducing the residual block, the network is allowed to learn the residual between the input and output, which effectively solves the problem of gradient disappearance or decreased representation ability that may occur in the deep neural network during training. In the super-resolution reconstruction task, residual learning can help the network better capture the high-frequency details of the image, thereby improving the quality of the reconstructed image.

[0109] B. The introduction of generative adversarial network (SRGAN) further improves the performance of super-resolution reconstruction.

[0110] SRGAN consists of a generator and a discriminator. Through adversarial training between the two, the generator can generate more realistic and higher-quality images. In the super-resolution reconstruction task, the generator's task is to learn the mapping relationship from low-resolution images to high-resolution images, while the discriminator is responsible for distinguishing the generated images from the real high-resolution images. Through adversarial training, the generator continuously optimizes the quality of the images it generates, making them closer to the real high-resolution images.

[0111] The SRGAN generative adversarial network is the main body of the network, which includes a generator G and a discriminator D. The input of the network is a high-resolution image, which is downsampled to a low-resolution image. The generator reconstructs the low-resolution image, and the discriminator distinguishes between the generated image and the input image. The generator and the discriminator confront each other and iterate continuously.

[0112] The generator D is mainly composed of Resnet (residual) blocks, so this generation network (generator) is also called SRResNet. Looking at the structure of SRResNet alone, it can be divided into low-level feature extraction, high-level feature extraction, deconvolution (transposed convolution) layer and the final CNN reconstruction layer. The introduction of residual learning is reflected in this part (residual block in the generator), which can prevent network degradation.

[0113] The discriminator D is a VGG network without a pooling layer. A binary classification is performed at the end of the network, which is actually to distinguish between the super-resolution image generated by the generator and the high-resolution image used as input.

[0114] In SRGAN, the objective function of the generative network is replaced by two customized image evaluation functions with medical significance. Although the perceptual loss can make the reconstructed image have high resolution perceived by the naked eye, it may introduce distortions and artifacts, affecting the medical accuracy of image super-resolution reconstruction. Therefore, the perceptual loss is not introduced for the reconstruction of OCT images.

[0115] C. Mask Generation Network (U-Net)

[0116] The target area of ​​the newly defined objective function has the characteristic of localization, that is, it only acts on the retinal nerve fiber layer area and the lamina cribrosa area, which requires the neural network to accurately identify the target area. The present invention generates masks corresponding to the nerve fiber layer area and the lamina cribrosa area through a U-Net network. It should be noted that: masks should be generated for the retinal nerve fiber layer area or the lamina cribrosa area, but the masks for the retinal nerve fiber layer area and the lamina cribrosa area are generated separately; in the process of generating the mask of the retinal nerve fiber layer area, the nerve fiber layer has a layered structure, and each layer of the nerve fiber layer corresponds to a mask.

[0117] The mask is essentially a feature map containing weights of different regions, which reflects the importance of different regions through weights. For OCT images of different patients or different positions of the same patient (OCT images are tomographic and have sequence characteristics, and the position corresponds to the order in the sequence), the positions of the nerve fiber layer and the lamina cribrosa are different, so each image should have its corresponding mask. During training, it is necessary to generate images through mask weighting to achieve the evaluation of the objective function for the local area.

[0118] The U-Net network uses an encoder-decoder structure and can effectively generate a mask corresponding to the input image. The training input is a high-resolution image processed by S100, and the output is a mask of the corresponding annotated target area (nerve fiber layer or lamina cribrosa). Here, "nerve fiber layer or lamina cribrosa" means that the masks of the retinal nerve fiber layer area and the lamina cribrosa area are generated separately, and the mask has the same dimension as the image. After the training is completed, the original high-resolution image and the image generated by the SRGAN network super-resolution (with the same number of channels, width, height and similar features as the high-resolution image of the training input) are used as input, and the U-Net network generates the corresponding mask, including the nerve fiber layer mask and the lamina cribrosa mask. The generated mask has the same dimension (number of channels, width, height) as the image. The generated mask will be used to implement the newly defined medically meaningful objective function for the evaluation of the local area (nerve fiber layer or lamina cribrosa).

[0119] In general, applying the SRGAN+U-Net network super-resolution reconstruction technology that introduces medical indicators as objective functions to enhance OCT images can not only restore more high-frequency detail information, making the reconstructed image visually closer to the real high-resolution image, but also maintain the medical authenticity of the image, providing strong support for the auxiliary diagnosis of early and middle-stage glaucoma.

[0120] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A training method for a generative model for super-resolution reconstruction of glaucoma fundus images, It is characterized in that The glaucoma fundus image super-resolution reconstruction generative model is a generator of a generative adversarial network. A high-resolution glaucoma fundus image is used as a reference image, and a low-resolution image obtained by downsampling the high-resolution glaucoma fundus image is used as the input of the generator. The generator is enabled to generate a glaucoma fundus image super-resolution reconstruction image through overall training of the generative adversarial network. During the training process of the generative adversarial network, the total loss function is obtained by using the loss function F, and the training of the generator is guided based on the total loss function. The loss function F is as follows: F=w 1 DCA_NFL(I ref ,I rec )+w 2 LC_ESM(I ref ,I rec ) Among them, w 1 、w 2 To adjust the weight, DCA_NFL(I ref ,I rec ) represents the image quality assessment loss, LC_ESM(I ref ,I rec ) represents edge similarity loss; A rec,n A ref , n represents the cross-sectional area of ​​the nerve fiber layer of the reference image and the reconstructed image, respectively, and N is the number of nerve fiber layers obtained by segmentation; E ref E rec are the binary edge maps of the lamina cribrosa region of the reference image and the reconstructed image respectively; |E ref ∩E rec | is the number of pixels where the binary edge of the lamina cribrosa region in the reference image overlaps with that in the reconstructed image; |E ref +E rec | is the sum of the number of binary edge pixels in the lamina cribrosa region in both the reference image and the reconstructed image, and is the total number of edge pixels in the two images.

2. A training method for a super-resolution reconstruction model for glaucoma fundus images according to claim 1, It is characterized in that The generative adversarial network is a SRGAN network.

3. The training method for generating a super-resolution reconstruction model for glaucoma fundus images according to claim 1, It is characterized in that During the training process of the generative adversarial network, the output of the generator in the generative adversarial network is guided by generating masks to determine the nerve fiber layer area and the cribriform plate area.

4. The training method for generating a super-resolution reconstruction model for glaucoma fundus images according to claim 3, It is characterized in that The mask generation method is implemented using the U-Net network.

5. The training method for generating a super-resolution reconstruction model for glaucoma fundus images according to claim 1, It is characterized in that During the training process of the generative adversarial network, the high-resolution glaucoma fundus image is a high-resolution glaucoma fundus image obtained by denoising the original glaucoma fundus image and then segmenting the OCT image based on Curvelet transform and watershed. In the OCT image segmentation process based on Curvelet transform and watershed, the glaucoma fundus image is first subjected to Curvelet transform, and then segmented using the OTSU algorithm.

6. The training method for generating a super-resolution reconstruction model for glaucoma fundus images according to claim 5, It is characterized in that After the image is segmented by the OTSU algorithm, it must undergo morphological processing to finally obtain a high-resolution glaucoma fundus image for training the generative adversarial network.

7. The training method for generating a super-resolution reconstruction model for glaucoma fundus images according to claim 5, It is characterized in that The denoising process of the original glaucoma fundus image is based on the Markov chain. In the denoising process of the Markov chain, the image q(y t-1 ), the image before denoising in the first step is the original glaucoma fundus image to be denoised, and the corresponding noise ε is first predicted using UNet T , and then use the following formula to denoise the current step; Among them, β t is the denoising coefficient; In the denoising process of the Markov chain, the denoising process of the current step is cycled to finally obtain a denoised high-resolution glaucoma fundus image.

8. The training method for generating a super-resolution reconstruction model for glaucoma fundus images according to claim 3, It is characterized in that In the process of segmentation using the OTSU algorithm, the calculation formula for the inter-class variance is as follows: c 2 =ω 1 oh 2 (r 1 -r 2 ) 2 W Among them, ω 1 ,ω 2 are the proportion of pixels below the threshold τ and the proportion of pixels above the threshold τ respectively; ρ 1 , 2 are the average grayscale values ​​of pixels below the threshold τ and the average grayscale values ​​of pixels above the threshold τ respectively; W is the weighting factor.

9. A super-resolution reconstruction method for glaucoma fundus images. It is characterized in that For the low-resolution glaucoma fundus image to be reconstructed, reconstruction is performed using a trained generator obtained by the training method for a glaucoma fundus image super-resolution reconstruction generation model described in any one of claims 1 to 8 to obtain a reconstructed super-resolution glaucoma fundus image.

10. The method for super-resolution reconstruction of glaucoma fundus images according to claim 9, It is characterized in that Before the low-resolution glaucoma fundus image to be reconstructed is input into the generator, it is necessary to denoise the low-resolution glaucoma fundus image to be reconstructed. The denoising process of the low-resolution glaucoma fundus image to be reconstructed is based on the Markov chain. In the denoising process of the Markov chain, the image q(y t-1 ), the image before denoising in the first step is the low-resolution glaucoma fundus image to be denoised, and the corresponding noise ε is first predicted using UNet T , and then use the following formula to denoise the current step; Among them, β t is the denoising coefficient; In the denoising process of the Markov chain, the denoising process of the current step is cycled to finally obtain a denoised low-resolution glaucoma fundus image.

Citation Information

Cited By

  • Glaucoma distinguishing detection method and system based on multi-modal fusion

    CN119313680A

  • Monocular depth estimation method and equipment based on improved depth distribution compensation and frequency domain feature fusion

    CN120031934A

  • Eye fundus image reconstruction method, system and device based on multi-dimensional enhancement and medium

    CN121366081A

  • Nasal mucosa microscopic three-dimensional reconstruction system based on optical coherence tomography

    CN122618167A

  • Nasal mucosa microscopic three-dimensional reconstruction system based on optical coherence tomography

    CN122618167B