A super-resolution reconstruction method for OCTA images based on deep neural network

By introducing the Fourier channel attention mechanism and deep neural network model in the super-resolution reconstruction of OCTA images, the problem of image quality improvement in the existing technology is solved, and higher resolution retinal choroidal blood flow imaging is achieved, and diagnostic accuracy is improved. and image reconstruction efficiency.

CN114298907BActive Publication Date: 2024-12-06UNIV OF SHANGHAI FOR SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111628819.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-28
Publication Date
2024-12-06
Estimated Expiration
2041-12-28

AI Technical Summary

Technical Problem

Existing super-resolution reconstruction techniques are difficult to effectively improve image quality in the field of medical imaging, especially fundus coherent phototogram angiography (OCTA), especially when blood flow is slower than the detectable range and image signal intensity. In the case of poor conditions, image blur and jagging will lead to image blur and serration, and the scope and effect of existing deep learning models in medical images are limited.

Method used

The super-resolution reconstruction method based on deep neural network is adopted to introduce the Fourier channel attention mechanism, and the image reconstruction effect is improved by adaptively adjusting the weight in the network, combining the U-net structure, GELU activation function and image denoising layer, and the image reconstruction effect is improved through Data preprocessing and transfer learning technologies solve the problem of insufficient data.

Benefits of technology

It significantly improves the resolution and diagnostic accuracy of OCTA images, can more clearly identify retinal choroidal blood flow movement information, improves the accuracy and convenience of diagnosis, and obtains high-quality reconstruction images more quickly and accurately than traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114298907B_ABST
    Figure CN114298907B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for super-resolution reconstruction of OCTA images based on a deep neural network, comprising: collecting OCTA images and preprocessing them to obtain an initial data set, dividing the initial data set into a training set and a test set; constructing a neural network model, training the neural network model based on the training set to obtain a first network model; testing the first network model based on the test set to obtain a second network model; and super-resolution reconstruction of OCTA images based on the second network model to obtain a target image. The reconstructed image of the present invention can help doctors identify retinal choroidal blood flow movement information with high resolution, clearly image the retinal choroidal microvessels of living tissue, and improve the accuracy and convenience of diagnosis.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of deep learning, and in particular relates to an OCTA image super-resolution reconstruction method based on a deep neural network. Background Art

[0002] Optical Coherence Tomography Angiography (OCTA) is a new non-invasive fundus detection technology that can identify the blood flow movement information of the retina and choroid with high resolution and image the microvascular circulation of the retina and choroid in living tissue. It has unique advantages in the management and follow-up of normal retinal and choroidal vascular changes and the detection of treatment effects.

[0003] Optical Coherence Tomography Angiography (OCTA) imaging principle: The principle of this technology is based on the existence of flowing blood cells in the fundus blood vessels. The movement of red blood cells in the vascular cavity is detected by measuring the changes in OCT signals of multiple scans of the same cross-section, and a complete three-dimensional retinal choroidal vascular image is obtained by combining the information of continuous en-face. Therefore, the relationship between the OCTA scanning signal and the direction of blood flow and the flow rate of blood flow will affect the signal intensity of OCTA. In the optic disc area, if the blood flow or tissue distribution is parallel to the direction of the OCT scanning signal, weak signals will appear on both OCTA and OCT. OCTA can only display blood flow signals within a limited speed range. The slowest detectable blood flow depends on the time interval between two consecutive OCTB-scan sequences. If the blood flow in the lesion is slower than the slowest detectable blood flow, it cannot be displayed in OCTA. Different from traditional angiography and OCT, OCTA adopts a new imaging mode, and the time required for a single blood flow imaging scan is 2-3s. Lens opacity, vitreous opacity or retinal disease can lead to poor OCTA scanning signal intensity and poor image quality. Therefore, how to efficiently improve image quality has become an urgent problem to be solved.

[0004] Super-resolution reconstruction technology (Super Resolution) refers to the reconstruction of a corresponding high-resolution image from an observed low-resolution image. Under certain conditions, it can overcome the inherent resolution limitation of the image system, thereby improving the resolution of the processed image. In the fundus coherent optical tomography vascular imaging system, super-resolution reconstruction technology can be used to improve image quality. In medical testing, it is often necessary to use tomography technology to identify and determine the precise location and details of the diseased body, such as the edge of the shadow, the size and position of the diseased body, etc. However, due to the limitations of hardware equipment and existing imaging technology, we cannot directly obtain high-definition images that can meet higher requirements through medical imaging systems. In the field of medical imaging, doctors often need the details of the image to help them make correct judgments, so super-resolution reconstruction technology plays a vital role in the field of medical imaging.

[0005] In 1955, Toraldo di Francia first clearly defined the concept of super-resolution in the field of optical imaging, which mainly refers to the process of using optical knowledge to restore data information beyond the diffraction limit. Around 1964, Harris and Goodman first proposed the concept of image super-resolution, which mainly refers to the process of synthesizing a single-frame image with richer detail information by using the method of extrapolating the spectrum. In 1984, based on the work of their predecessors, Tsai and Huang et al. first proposed a method of reconstructing a high-resolution image using multiple frames of low-resolution images, and super-resolution reconstruction technology began to receive widespread attention and research from academia and industry. Super-resolution reconstruction technology can be divided into two categories: reconstructing a high-resolution image from multiple low-resolution images and reconstructing a high-resolution image from a single high-resolution image. Traditional super-resolution reconstruction methods usually use interpolation methods: interpolation-based methods regard each image as a point on the image plane, so the estimation of super-resolution images can be regarded as a process of fitting unknown pixel information on the plane using known pixel information, which is usually completed by a predefined transformation function or interpolation kernel. The interpolation-based method is simple to calculate and easy to understand, but it also has some obvious defects. First, it assumes that the change of pixel grayscale value is a continuous and smooth process, but in fact this assumption is not completely true. Second, in the reconstruction process, the super-resolution image is calculated only based on a pre-defined conversion function without considering the image degradation model, which often causes the restored image to appear blurry and jagged.

[0006] With the continuous development of artificial intelligence and computer hardware, Hinton et al. proposed the concept of deep learning in 2006, which aims to extract high-level abstract features of data by using multi-layer nonlinear transformation. With its powerful fitting ability, deep learning has begun to emerge in various fields, especially in the field of image and vision, where convolutional neural networks have made great progress. This has also led to more and more researchers trying to introduce deep learning into the field of super-resolution reconstruction. In 2014, Dong et al. first applied deep learning to the field of image super-resolution reconstruction. They used a three-layer convolutional neural network to learn the mapping relationship between low-resolution images and high-resolution images. Since then, a wave of deep learning has been set off in the field of super-resolution reconstruction. Through the current mainstream research, it is found that the superior areas of existing super-resolution neural network models are mainly concentrated in low-complexity biological structures and imaging conditions that increase the resolution by two times. How to further expand the scope of application of convolutional neural networks in medical images and improve super-resolution imaging and reconstruction effects has become an urgent problem for researchers to solve. Summary of the invention

[0007] The present invention aims to address the deficiencies in the prior art and provide a super-resolution neural network model based on deep learning, introduce the Fourier channel attention mechanism into the neural network model, and improve the effects of super-resolution imaging and reconstruction.

[0008] To achieve the above object, the present invention provides the following solution: a method for super-resolution reconstruction of OCTA images based on a deep neural network, comprising:

[0009] Acquiring OCTA images, preprocessing the OCTA images to obtain an initial data set, and dividing the initial data set into a training set and a test set;

[0010] Constructing a neural network model, and training the neural network model based on the training set to obtain a first network model;

[0011] Testing the first network model based on the test set to obtain a second network model;

[0012] The OCTA image is super-resolution reconstructed based on the second network model to obtain a target image.

[0013] The process of preprocessing the OCTA image includes cropping and flipping the OCTA image;

[0014] The cropping is to crop the OCTA image to a size of 512x512;

[0015] The flipping is to perform 90°, 180° and 270° flipping operations on the cropped image to obtain the initial data set.

[0016] The process of constructing the neural network model includes:

[0017] On the basis of U-net, the two 3x3 convolutions are replaced with 3x3 convolutions and 7x7 convolutions, which are then operated in parallel with the 5x5 convolutions. A 1x1 convolution layer is introduced, and the simple skip connection in the traditional U-net network is replaced by a residual connection, which first passes through the convolution layer chain with residual connections and then connects to the decoder features.

[0018] Constructing the neural network model also includes obtaining a first activation function GELU;

[0019] The first activation function GELU is obtained by introducing a random regularization idea into the activation function during activation;

[0020] The first activation function GELU expression is:

[0021]

[0022] in,

[0023] It also includes adding a Fourier channel attention mechanism at the beginning of the decoder, adaptively adjusting the weights when propagating between networks according to the frequency domain differences between different features, and obtaining a hierarchical mapping of the network to the low-resolution image.

[0024] The Fourier channel attention mechanism consists of four identical residual groups, each of which consists of four Fourier channel attention modules and a skip link.

[0025] Training the neural network model based on the training set also includes sending the residual group to a convolution layer activated by a first activation function GELU, adding the decoder to an image denoising layer, and obtaining a first grayscale image based on the convolution layer and the Sigmoid function.

[0026] Obtaining the first network model also includes building a training model using the tensorflow deep learning framework.

[0027] The process of testing the first network model based on the test set includes inputting the test set into the first network model, obtaining a peak signal-to-noise ratio of the test set, performing quantitative evaluation based on the peak signal-to-noise ratio, and obtaining an evaluation index value.

[0028] The present invention discloses the following technical effects:

[0029] 1. The present invention provides an OCTA image super-resolution reconstruction method based on a deep neural network, which can help doctors identify retinal choroidal blood flow movement information with high resolution through the reconstructed image, clearly image the retinal choroidal microvessels of living tissue, and improve the accuracy and convenience of diagnosis;

[0030] 2. The present invention improves the neural network model by introducing the Fourier attention mechanism. By utilizing the frequency domain differences between different features, the weights can be adaptively adjusted when propagating between networks. This mechanism enables the network to have more accurate hierarchical mapping for low-resolution images and improve the reconstruction effect.

[0031] 3. The present invention enhances the data set image in the preprocessing stage, which can provide high-quality data input for the neural network and effectively improve the accuracy of model training. Compared with the existing methods, this method can obtain the reconstructed image more quickly and accurately;

[0032] 4. Eye images are an important means and channel for diagnosing eye diseases, so the number of data sets available for training is very limited. The present invention effectively solves the problem that a large number of data sets are required for training neural networks by using transfer learning technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] like Figure 1 As shown, the present invention provides an OCTA image super-resolution reconstruction method based on a deep neural network, comprising:

[0038] Acquiring OCTA images, preprocessing the OCTA images to obtain an initial data set, and dividing the initial data set into a training set and a test set;

[0039] Constructing a neural network model, and training the neural network model based on the training set to obtain a first network model;

[0040] Testing the first network model based on the test set to obtain a second network model;

[0041] The OCTA image is super-resolution reconstructed based on the second network model to obtain a target image.

[0042] The process of preprocessing the OCTA image includes cropping and flipping the OCTA image;

[0043] The cropping is to crop the OCTA image to a size of 512x512;

[0044] The flipping is to perform 90°, 180° and 270° flipping operations on the cropped image to obtain the initial data set.

[0045] The process of constructing the neural network model includes:

[0046] On the basis of U-net, the two 3x3 convolutions are replaced with 3x3 convolutions and 7x7 convolutions, which are then operated in parallel with the 5x5 convolutions. A 1x1 convolution layer is introduced, and the simple skip connection in the traditional U-net network is replaced by a residual connection, which first passes through the convolution layer chain with residual connections and then connects to the decoder features.

[0047] Constructing the neural network model also includes obtaining a first activation function GELU;

[0048] The first activation function GELU is obtained by introducing a random regularization idea into the activation function during activation;

[0049] The first activation function GELU expression is:

[0050]

[0051] in,

[0052] It also includes adding a Fourier channel attention mechanism at the beginning of the decoder, adaptively adjusting the weights when propagating between networks according to the frequency domain differences between different features, and obtaining a hierarchical mapping of the network to the low-resolution image.

[0053] The Fourier channel attention mechanism consists of four identical residual groups, each of which consists of four Fourier channel attention modules and a skip link.

[0054] Training the neural network model based on the training set also includes sending the residual group to a convolution layer activated by a first activation function GELU, adding the decoder to an image denoising layer, and obtaining a first grayscale image based on the convolution layer and the Sigmoid function.

[0055] Obtaining the first network model also includes building a training model using the tensorflow deep learning framework.

[0056] The process of testing the first network model based on the test set includes inputting the test set into the first network model, obtaining a peak signal-to-noise ratio of the test set, performing quantitative evaluation based on the peak signal-to-noise ratio, and obtaining an evaluation index value.

[0057] Embodiment 1

[0058] like Figure 1 As shown, the OCTA image super-resolution reconstruction method based on deep neural network provided by the present invention comprises the following steps:

[0059] 1) Data acquisition: OCTA images are collected as a data set. In the process of collecting OCTA image data of patients' eyes, the data set is manually labeled by expert doctors. The collected image data set is divided into a training set and a test set in a ratio of 7:3;

[0060] 2) Data preprocessing: In order to train the model more fully, the data set needs to be preprocessed. The data set images acquired by the OCTA imaging system are cropped to a size of 512x512, and then the images are flipped 90°, 180°, and 270° in sequence. Through the above operations, the number of data sets can be increased to four times the number of original data sets, effectively solving the problem that the neural network requires a large number of data sets when training. Finally, the translation operation can obtain the final enhanced data set; by enhancing the data set images, high-quality data input is provided for the neural network, which effectively improves the accuracy of model training and can obtain reconstructed images more quickly and accurately;

[0061] 3) Network construction: Construct a neural network model. Based on U-net, replace two 3x3 convolutions with a 3x3, 7x7 and 5x5 convolution in parallel, introduce a 1x1 convolution layer, and use residual links to replace the simple skip connection in the traditional U-net network. Instead of simply connecting feature maps from the encoder to the decoder, they first pass through a chain of convolutional layers with residual connections and then connect to the decoder features, which can effectively help the model understand some other spatial information. Introduce a Fourier channel attention module in the neural network, which consists of a 3x3 convolution layer, a GELU activation function, a Fourier transform layer, and a global average pooling layer; according to the frequency domain differences between different features, the weights are adaptively adjusted when propagating between networks to obtain the network's hierarchical mapping of low-resolution images. The architecture starts with a convolution layer and an activation function GELU. Compared with other common activation functions, the activation function GELU introduces the idea of ​​random regularization in activation. It is a probabilistic description of neurons, not easy to saturate and has random factors.

[0062] The formula for the activation function GELU is as follows:

[0063]

[0064] in,

[0065] Then, the Fourier channel attention mechanism is added at the beginning of the encoder. The mechanism consists of four identical residual groups, each of which consists of four Fourier channel attention modules and a jump link. By utilizing the frequency domain differences between different features, the weights can be adaptively adjusted when propagating between networks. This mechanism enables the network to have a more accurate hierarchical mapping for low-resolution images. The residual group is denoted as RG:

[0066] Rg(x)=x+FCAB (4) (x)

[0067] Among them, FCAB (n) (x) = FCAB(FCAB(...FCAB(x)))

[0068] Let x denote the input feature map of the residual group. In each Fourier attention channel module, the feature map will be rescaled as follows:

[0069]

[0070] Where y = GELU Conv{GELUConv{x}}

[0071]

[0072] The output of the last residual group will be sent to the convolution layer activated by the GELU activation function. At the same time, we added an image denoising layer in the decoder, and finally passed through a convolution layer and a Sigmoid function to obtain the super-resolution reconstructed grayscale image.

[0073] 4) Model training: Use the tensorflow deep learning framework to build a training model.

[0074] 5) Model evaluation: Peak signal to noise ratio (PSNR) is used to quantitatively evaluate the quality of SR. The higher the value of the indicator, the closer the pixel value of the reconstructed result is to the standard:

[0075]

[0076] in,

[0077] MSE represents the mean square error between the current image X and image Y, where H and W are the height and width of the image respectively.

[0078] The image super-resolution reconstruction method of the present invention obtains a super-resolution reconstructed target image by sequentially passing a low-resolution OCTA image through a 3x3 convolution layer, an excitation function, a downsampling module, an upsampling module, a Fourier channel attention module, and a Sigmoid function. The reconstructed image can help doctors identify retinal choroidal blood flow movement information with high resolution, clearly image the retinal choroidal microvessels of living tissue, and improve the accuracy and convenience of diagnosis.

[0079] The embodiments described above are only descriptions of the preferred modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A super-resolution reconstruction method for OCTA images based on deep neural network, It is characterized in that include: Acquiring OCTA images, preprocessing the OCTA images to obtain an initial data set, and dividing the initial data set into a training set and a test set; Constructing a neural network model, and training the neural network model based on the training set to obtain a first network model; Testing the first network model based on the test set to obtain a second network model; Performing super-resolution reconstruction on the OCTA image based on the second network model to obtain a target image; The process of constructing the neural network model includes: On the basis of U-net, the two 3x3 convolutions are replaced with 3x3 convolutions and 7x7 convolutions, which are parallelized with the 5x5 convolutions. A 1x1 convolution layer is introduced, and the simple skip connection in the traditional U-net network is replaced with a residual connection. The convolution layer chain with residual connections is first passed through, and then connected to the decoder features. It also includes adding a Fourier channel attention mechanism at the beginning of the decoder, adaptively adjusting the weights when propagating between networks according to the frequency domain differences between different features, and obtaining a hierarchical mapping of the network to the low-resolution image; The Fourier channel attention mechanism consists of four identical residual groups, each of which consists of four Fourier channel attention modules and a skip link; Training the neural network model based on the training set also includes sending the residual group to a convolution layer activated by a first activation function GELU, adding the decoder to an image denoising layer, and obtaining a first grayscale image based on the convolution layer and the Sigmoid function.

2. The OCTA image super-resolution reconstruction method based on deep neural network according to claim 1, It is characterized in that The process of preprocessing the OCTA image includes cropping and flipping the OCTA image; The cropping is to crop the OCTA image to a size of 512x512; The flipping is to perform 90°, 180° and 270° flipping operations on the cropped image to obtain the initial data set.

3. The OCTA image super-resolution reconstruction method based on deep neural network according to claim 1, It is characterized in that Constructing the neural network model also includes obtaining a first activation function GELU; The first activation function GELU is obtained by introducing a random regularization idea into the activation function during activation; The first activation function GELU expression is: in, .

4. The OCTA image super-resolution reconstruction method based on deep neural network according to claim 1, It is characterized in that Obtaining the first network model also includes building a training model using the tensorflow deep learning framework.

5. The OCTA image super-resolution reconstruction method based on deep neural network according to claim 1, It is characterized in that The process of testing the first network model based on the test set includes inputting the test set into the first network model, obtaining a peak signal-to-noise ratio of the test set, performing quantitative evaluation based on the peak signal-to-noise ratio, and obtaining an evaluation index value.