Image processing apparatus and method for improving image quality of retinal fundus image

Abstract

An image processing apparatus for improving the image quality of a retinal fundus image according to one aspect of the present invention comprises: a memory for storing an image quality improvement program; and a processor for executing the image quality improvement program, wherein the image quality improvement program improves the image quality of an input fundus image through a deep neural network of a U-net structure including an encoder unit and a decoder unit, the encoder unit includes at least one down-sampling layer and an attention block connected to each down-sampling layer, the decoder unit includes at least one up-sampling layer and an attention block connected to each up-sampling layer, and the deep neural network is trained to improve a deterioration region included in the input fundus image.

Description

Image processing device and method for improving the image quality of retinal fundus images

[0001] The present invention relates to an image processing device and method for improving the image quality of a retinal fundus image.

[0002] Retinal fundus imaging is a crucial ophthalmic examination tool for diagnosing and monitoring retinal diseases. It is reliable, noninvasive, and relatively inexpensive. This technology allows clinicians to observe the retina in detail through high-quality, high-resolution images. Retinal fundus photography is one of the most fundamental imaging modalities and is used to diagnose major retinal diseases, such as age-related macular degeneration and diabetic retinopathy. As life expectancy increases globally, the prevalence of chronic age-related ophthalmic diseases is on the rise, and the demand for high-quality fundus photography is expected to increase accordingly.

[0003] While retinal cameras used in eye examinations offer cutting-edge technology for fundus imaging, the quality of each fundus image can vary depending on the environment, operator, or patient. For example, motion blur may occur if the patient moves, eye blinking may result in occlusions in the image, or lighting may be insufficient. This can make it difficult for clinicians to make an effective diagnosis and, consequently, grade the fundus image. This necessitates re-examination of the patient to obtain accurate fundus imaging results, resulting in unnecessary costs and delays.

[0004] The present invention aims to propose a method for improving the image quality of low-quality fundus images. In this regard, a novel image processing method is proposed by improving upon a previously known deep neural network-based image quality improvement method.

[0005] In this regard, prior art literature discloses Korean Patent Publication No. 10-2022-0164953 (Title of invention: Retinal blood vessel detection system using deep learning).

[0006] In order to solve the above-mentioned problems, the present invention provides an image processing device and method for improving the image quality of a fundus image using a deep neural network.

[0007] However, the technical tasks that this embodiment seeks to accomplish are not limited to the technical tasks described above, and other technical tasks may exist.

[0008] As a technical means for achieving the above-described technical task, an image processing device for improving the image quality of a retinal fundus image according to an aspect of the present invention comprises: a memory storing an image quality improvement program; and a processor executing the image quality improvement program, wherein the image quality improvement program improves the image quality of an input fundus image through a deep neural network having a U-net structure including an encoder unit and a decoder unit, wherein the encoder unit includes at least one down-sampling layer and an attention block connected to each down-sampling layer, and the decoder unit includes at least one up-sampling layer and an attention block connected to each up-sampling layer, and the deep neural network is trained to improve a deteriorated area included in an input fundus image.

[0009] According to another aspect of the present invention, an image processing method for improving the image quality of a retinal fundus image comprises the steps of: receiving an input fundus image as a target of image quality improvement; and inputting the input fundus image into a deep neural network and outputting an output fundus image, wherein the deep neural network has a U-net structure including an encoder unit and a decoder unit, the encoder unit includes at least one down-sampling layer and an attention block connected to each down-sampling layer, the decoder unit includes at least one up-sampling layer and an attention block connected to each up-sampling layer, and the deep neural network is trained to improve a deteriorated area included in the input fundus image.

[0010] According to the solution to the aforementioned problem, a new configuration is provided in which attention blocks are respectively connected to the up-sampling layer and the down-sampling layer in a deep neural network with a U-net structure, thereby improving low-quality fundus images to high-quality ones.

[0011] FIG. 1 is a block diagram illustrating the configuration of an image processing device for improving the image quality of a retinal fundus image according to one embodiment of the present invention.

[0012] Figure 2 illustrates the configuration of a deep neural network of a picture quality improvement program according to one embodiment of the present invention.

[0013] Figure 3 illustrates a detailed configuration of an encoder block according to one embodiment of the present invention.

[0014] Figure 4 illustrates a detailed configuration of a decoder block according to one embodiment of the present invention.

[0015] Figure 5 illustrates a detailed configuration of an attention block according to one embodiment of the present invention.

[0016] Figure 6 is a flowchart illustrating an image processing method according to one embodiment of the present invention.

[0017] FIG. 7 is a diagram illustrating a test result of an image processing device according to one embodiment of the present invention.

[0018] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement them. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, irrelevant parts have been omitted for clarity of description, and similar reference numerals have been used throughout the specification to indicate similar elements.

[0019] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where the parts are "directly connected" but also the cases where the parts are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes other components, unless otherwise stated.

[0020] In this specification, the term 'unit' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be realized by using two or more pieces of hardware, and two or more units may be realized by one piece of hardware. Meanwhile, the '~ unit' is not limited to software or hardware, and the '~ unit' may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. Therefore, as an example, the '~ unit' includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and '~sub-units' may be combined into a smaller number of components and '~sub-units' or further separated into additional components and '~sub-units'. Furthermore, the components and '~sub-units' may be implemented to recycle one or more CPUs within the device.

[0021] The image processing device mentioned below can be implemented as a computer or portable terminal that can connect to a server or other terminal via a network. Here, the computer includes, for example, a notebook, desktop, or laptop equipped with a web browser, and the portable terminal can include, for example, a wireless communication device that guarantees portability and mobility, and can include all kinds of handheld-based wireless communication devices such as various mobile communication-based terminals, smartphones, tablet PCs, etc. In addition, the "network" can be implemented as a wired network such as a local area network (LAN), a wide area network (WAN), or a value-added network (VAN), or all kinds of wireless networks such as a mobile radio communication network or a satellite communication network.

[0022] FIG. 1 is a block diagram illustrating the configuration of an image processing device for improving the image quality of a retinal fundus image according to one embodiment of the present invention.

[0023] The image processing device (100) includes a processor (110) and a memory (120), and may further include a communication module (130) and a database (140). In addition, the image processing device (100) may be connected to a device (not shown) such as a specialized medical device or a camera that captures a fundus image. The image processing device (100) applies an image quality improvement program to an input fundus image to improve the image quality and output the image.

[0024] As such, the image processing device (100) may also be implemented in the form of a server that provides fundus images with improved image quality. In this case, when the image processing device (100) functions as a server, it may operate in a cloud computing service model such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). Furthermore, the image processing device (100) may be constructed in the form of a private cloud, a public cloud, or a hybrid cloud.

[0025] The memory (120) records an image quality improvement program. The image quality improvement program improves the image quality of an input fundus image through a deep neural network of a U-net structure including an encoder unit and a decoder unit, and the deep neural network is trained to improve a deteriorated area included in the input fundus image. At this time, the encoder unit includes at least one down-sampling layer and an attention block connected to each down-sampling layer, and the decoder unit includes at least one up-sampling layer and an attention block connected to each up-sampling layer.

[0026] In addition, the memory (120) performs the function of temporarily or permanently storing data processed by the processor (110). Here, the memory (120) may include a volatile storage media or a non-volatile storage media, but the scope of the present invention is not limited thereto.

[0027] The processor (110) executes an image quality improvement program stored in the memory (120). In addition, the processor (110) performs various control operations for the operation of the image processing device (100). The processor (110) may refer to a data processing device built into hardware, which has a physically structured circuit to perform a function expressed by a code or command included in a program, for example. As an example of a data processing device built into hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), and the like may be included, but the scope of the present invention is not limited thereto.

[0028] The communication module (130) may include one or more components that enable transmission and reception of various types of data with an external computing device or camera. For example, the communication module (130) may receive fundus images or other various types of data, and transmit fundus images with improved image quality to another computing device. The communication module (130) may be a device including the hardware and software necessary to transmit and receive signals, such as control signals or data signals, with other network devices via wired or wireless connections.

[0029] The database (140) can manage various data required for executing an image quality improvement program. The database (140) can store various fundus image data as learning data required for constructing a deep neural network, or can store fundus images with improved image quality output by a deep neural network.

[0030] Now, let's take a look at the detailed configuration of the image quality improvement program.

[0031] Figure 2 illustrates the configuration of a deep neural network of a picture quality improvement program according to one embodiment of the present invention.

[0032] A deep neural network (200) according to one embodiment of the present invention has a U-net structure including an encoder unit (210) and a decoder unit (230). The deep neural network (200) is constructed based on a convolutional neural network (CNN). The encoder unit (210) includes at least one down-sampling layer (211, 213, 215, 217) and an attention block (212, 214, 216, 218) connected to each down-sampling layer (211, 213, 215, 217). In addition, the decoder unit (230) includes at least one up-sampling layer (231, 233, 235, 237) and attention blocks (232, 234, 236, 238) connected to each up-sampling layer. The attention blocks (232, 234, 236, 238) are components that process to pay more attention to specific features in CNN, etc., and this attention processing technique itself corresponds to a conventionally known component. At this time, the number of down-sampling layers included in the encoder unit (210) or the number of up-sampling layers included in the decoder unit (230) can be changed according to the user's selection. However, the encoder unit (210) and the decoder unit (230) are configured to have the same number of down-sampling layers and up-sampling layers, respectively. In addition, an attention block is connected to each down-sampling layer and up-sampling layer.

[0033] Meanwhile, each downsampling layer (211, 213, 215, 217) performs various pooling processes, such as max pooling, after performing a convolution operation on the filter on the feature information received in the previous step. In addition, each upsampling layer (231, 233, 235, 237) performs a transpose convolution on the feature information received in the previous step. Additionally, each down-sampling layer (211, 213, 215, 217) and each up-sampling layer (231, 233, 235, 237) may have almost the same structure. For example, if each down-sampling layer (211, 213, 215, 217) has a 3 Х 3conv structure, the up-sampling layer (231, 233, 235, 237) may have a 3 Х 3 deconv structure.

[0034] Meanwhile, each attention block connected to the down-sampling layer and up-sampling layer that are in a symmetrical or facing relationship is connected in the form of a skip connection. For example, as shown in the drawing, when the encoder unit (210) and the decoder unit (230) each include four sampling layers, except for the down-sampling layer (217) located at the very end in the encoder unit (210), the attention block (212) of the first down-sampling layer (211) and the attention block (236) of the third up-sampling layer (235) located at the very front are connected by a skip connection. In addition, the attention block (214) of the second down-sampling layer (213) and the attention block (234) of the second up-sampling layer (233) are connected by a skip connection. Additionally, the attention block (216) of the third down-sampling layer (215) and the attention block (232) of the first up-sampling layer (231) are connected by a skip connection. Through this skip connection, feature information obtained from each layer of the preceding encoder unit (210) can be concatenated into each layer of the decoder unit (230).

[0035] Meanwhile, in the conventional Unet structure, downsampling layers and upsampling layers are connected via skip connections. However, in the present invention, attention blocks are additionally placed for each sampling layer, and skip connections are connected between the attention blocks. Furthermore, each attention block includes layers that perform multiple convolution operations (e.g., 1x1, 3x3, 5x5, 7x7, etc.).

[0036] The rationale behind this configuration is as follows. Real-world fundus images contain a complex and scattered array of degradation regions. Identifying the type, size, and location of degradation regions within a single image is crucial. To effectively address this, multiple convolutional operations within each attention block are applied, enabling efficient processing of degradation region features through operation-wise attention.

[0037] FIG. 3 illustrates a detailed configuration of an encoder block according to one embodiment of the present invention, FIG. 4 illustrates a detailed configuration of a decoder block according to one embodiment of the present invention, and FIG. 5 illustrates a detailed configuration of an attention block according to one embodiment of the present invention.

[0038] The encoder block illustrated in FIG. 3 illustrates a configuration in which one downsampling layer (e.g., 3x3Conv) and an attention block are connected.

[0039] The block shown with a dotted line represents an attention branch, which may sequentially include a batch normalization layer, an activation function layer (LreLU), an attention block, and an activation function layer (LreLU).

[0040] Additionally, the decoder block illustrated in Fig. 4 illustrates a configuration in which one upsampling layer (e.g., 3x3DeConv) and an attention block are connected. Similarly to Fig. 3, the block illustrated with a dotted line represents an attention branch, which may sequentially include a batch normalization layer, an activation function layer (LreLU), an attention block, and an activation function layer (LreLU).

[0041] For reference, the width and height of the feature map in Fig. 3 are 2 in the image. 4 downsampled as much as possible, and after the encoding part of the first half of the network, the feature dimension is 2. 10 This can be. For example, the size of the latent feature derived by downsampling from an input image of size 320Х320Х3 can be 20Х20Х1024. In addition, when the latent feature is upsampled and reconstructed in the decoding part of the latter half of the network, an output image of size 320Х320Х3 with the same size as the input image is output.

[0042] Referring to Fig. 5, each attention block can apply multiple operations in parallel. For example, each attention block can process 1 Х 1conv, 3 Х 3conv, 5 Х 5conv, 7 Х 7conv, or 3 Х 3maxpool operations in parallel. This allows the attention block to adaptively apply operation-wise attention.

[0043] Additionally, the attention block applies a channel-specific attention layer to calculate attention weights that indicate the importance of each operation. Each attention layer calculates attention weights through a 3-layer MLP based on the channel-specific average of the input feature map, and searches for the optimal operation to be used in the encoder and decoder blocks by considering various factors such as the size of the feature map, degradation factor, severity of degradation, and layer depth.

[0044] In the attention block (l), the attention weight (Al ) can be defined as in mathematical expression 1.

[0045] [Mathematical Formula 1]

[0046]

[0047] represents the learnable matrix, |O| represents the number of operations in the attention layer, and F r represents the ReLU activation function, and C l Silver input (X l ) represents the spatial average per channel, and can be defined as in mathematical expression 2.

[0048] [Equation 2]

[0049]

[0050] H is the input feature map (X l ) represents the height of the input feature map (X l ) represents the width of the input feature map (X l ) represents the channel. In the present invention, the channel-specific average is used as the input of the channel-specific attention layer, because the absolute intensity of the pixel map of the input feature has a significant influence on determining the deterioration factor and severity.

[0051] Then, vector (A) is It is normalized as in Equation 3 so that the sum of the attention weights becomes 1.

[0052] [Equation 3]

[0053]

[0054] And, Z l silver Wow Y l is the result of the element-wise product of Y l Silver Y l =O(X l ), it represents the result of applying each operation of the operation set to the input feature map of the layer.

[0055] [Equation 4]

[0056]

[0057] represents element-wise multiplication.

[0058] Input feature map (X l ) is Z l , and are concatenated to maintain the knowledge learned in the previous layer. This connection can also be interpreted as a residual connection between the input and output of the layer, which allows the gradient to be propagated to the input of the layer through backpropagation. Finally, a 1 Х 1conv operation can be placed at the end of the layer to adjust the channels of the output feature map of the attention block. And, the output of the attention block (l) (S l ) can be calculated as follows.

[0059] [Equation 5]

[0060]

[0061] [Equation 6]

[0062]

[0063] |O| represents the number of operations in the attention layer, and F c represents the 1 Х 1conv operation, represents the channel-wise concatenation of two matrices.

[0064] Meanwhile, the learning function used to learn the deep neural network of the present invention may use the function of mathematical formula 7, but this is exemplary and the present invention is not limited thereto.

[0065] [Equation 7]

[0066]

[0067] Here, y represents the output of the deep neural network, represents the reference image included in the learning data, and N batch represents the number of images in a mini-batch, and W net represents the weight parameters of the deep neural network. The first term in Equation 7 represents the pixel-by-pixel difference to supervise the output of the deep neural network to be similar to the reference image (or high-quality image), and the second term represents the L2 norm for the trainable network weights, which is a commonly used regularization term. In the present invention, the L1 distance is used for the pixel-by-pixel difference, but unlike other works, the L2 distance may excessively penalize pixel values ​​with uneven illumination. This is because the training data set may include many pairs of dark low-quality images and bright high-quality images. Meanwhile, the parameter λ controls the relative importance between the first and second terms in Equation 7, and can be set to, for example, λ = 0.1.

[0068] In addition, the training data used to train the deep neural network of the present invention can be data that matches low-resolution fundus images and high-resolution fundus images for each patient. To verify the deep neural network of the present invention, 1068 pairs of low-resolution images (LQ) and high-resolution images (HQ) obtained from a specialized hospital were used as training data. In this case, the low-resolution fundus images and high-resolution fundus images were taken of the same patient, and the low-resolution fundus images can be fundus images before eye disease surgery, and the high-resolution fundus images can be fundus images after eye disease surgery.

[0069] In addition, additional augmentation processing can be performed on the learning data collected in this way. That is, preprocessing can be performed to rotate each image by a certain angle (such as 90 degrees, 180 degrees, or 270 degrees). Alternatively, low-quality images (L I ) and high definition video (H I ) can be preprocessed by performing linear interpolation as in mathematical expression 8.

[0070] [Equation 8]

[0071]

[0072] Here, λ is a parameter between 1 and 0 that controls the degree of linear interpolation.

[0073] Additionally, preprocessing can be performed to perform blur or haze processing on each image.

[0074] That is, as in mathematical expression 9, high-definition image (H I ) by performing Gaussian blur processing on low-quality images (L I ) can be created.

[0075] [Equation 9]

[0076]

[0077] K represents the Gaussian kernel for convolution.

[0078] And, as in mathematical expression 10, high-definition image (H I ) to perform haze processing on low-quality images (L I ) can be created.

[0079] [Equation 10]

[0080]

[0081] t(x) represents the transmission map, and A represents the atmospheric light vector in the RGB domain. If the transmission map is homogeneous, t(x) can be defined as in Equation 11.

[0082] [Equation 11]

[0083]

[0084] β represents the medium extinction coefficient, and d(x) represents the depth between the object and the camera.

[0085] Additionally, data can be augmented through preprocessing that adjusts the brightness of each image.

[0086] In the mathematical expression 12 below, by performing γ correction on a high-quality image, a low-quality image with uneven illumination can be generated.

[0087] [Equation 12]

[0088]

[0089] Figure 6 is a flowchart illustrating an image processing method according to one embodiment of the present invention.

[0090] First, the image processing device (100) receives an input fundus image to be improved in image quality (S110).

[0091] The input fundus image can be directly input through the IO interface of the image processing device (100) or received from an external computing device through a communication module (130). The input fundus image can include at least one deteriorated area, and the image processing device (100) processes the image to perform improvement processing on the deteriorated area.

[0092] Next, the image processing device (100) inputs the input fundus image into a deep neural network and outputs an output fundus image (S120).

[0093] At this time, the deep neural network (200) is configured as a U-net structure including an encoder unit (210) and a decoder unit (230), as described above. The encoder unit (210) includes at least one down-sampling layer and an attention block connected to each down-sampling layer, and the decoder unit (230) includes at least one up-sampling layer and an attention block connected to each up-sampling layer. Through this deep neural network, the deteriorated area included in the input fundus image can be improved, thereby improving a low-quality fundus image into a high-quality fundus image.

[0094] FIG. 7 is a diagram illustrating a test result of an image processing device according to one embodiment of the present invention.

[0095] (a) shows the input low-quality image, and (b) shows the output high-quality image. In particular, it can be seen that the image quality of the lesion area highlighted by the square box is improving.

[0096] A method according to one embodiment of the present invention may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Computer-readable media may also include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0097] Although the methods and systems of the present invention have been described with respect to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

[0098] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will readily appreciate that the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner.

[0099] The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

Claims

1. In an image processing device for improving the image quality of a retinal fundus image, Memory storing the image quality improvement program; and Including a processor that executes the above image quality improvement program, The above image quality improvement program improves the image quality of the input fundus image through a deep neural network of U-net structure including an encoder section and a decoder section. The above encoder unit includes at least one down-sampling layer and an attention block connected to each down-sampling layer, The above decoder unit includes at least one up-sampling layer and an attention block connected to each up-sampling layer, An image processing device, wherein the deep neural network is trained to improve a deteriorated area included in an input fundus image.

2. In paragraph 1, An image processing device, wherein the encoder and decoder each include the same number of down-sampling layers and up-sampling layers, and each attention block connected to a down-sampling layer and an up-sampling layer that are symmetrical to each other is connected in the form of a skip connection.

3. In paragraph 1, An image processing device wherein the above attention block performs operation-wise attention by performing multiple convolution operations in parallel.

4. In paragraph 1, The above deep neural network is an image processing device that is trained based on learning data that matches low-quality fundus images and high-quality fundus images for each patient.

5. In paragraph 4, An image processing device wherein the above low-resolution fundus image and the above high-resolution fundus image were taken of the same patient, wherein the low-resolution fundus image is a fundus image of the patient before eye disease surgery, and the above high-resolution fundus image is a fundus image after eye disease surgery.

6. In paragraph 4, An image processing device, wherein the learning data includes data augmented through preprocessing that rotates each image by a predetermined angle, preprocessing that performs linear interpolation on low-quality and high-quality images, preprocessing that performs blur or haze processing on each image, or preprocessing that adjusts the brightness of each image.

7. In an image processing method for improving the image quality of a retinal fundus image, A step of receiving an input fundus image to be subject to image quality improvement; and Including a step of inputting the above input fundus image into a deep neural network and outputting an output fundus image, The above deep neural network consists of a U-net structure including an encoder section and a decoder section. The above encoder unit includes at least one down-sampling layer and an attention block connected to each down-sampling layer, The above decoder unit includes at least one up-sampling layer and an attention block connected to each up-sampling layer, An image processing method, wherein the above deep neural network is trained to improve a deteriorated area included in an input fundus image.

8. In paragraph 7, An image processing method, wherein the encoder and decoder each include the same number of down-sampling layers and up-sampling layers, and each attention block connected to a down-sampling layer and an up-sampling layer that are symmetrical to each other is connected in the form of a skip connection.

9. In paragraph 7, An image processing method, wherein the above attention block performs operation-wise attention by performing multiple convolution operations in parallel.

10. In paragraph 7, The above deep neural network is an image processing method, wherein the deep neural network is trained based on learning data that matches low-quality fundus images and high-quality fundus images for each patient.

11. In paragraph 10, An image processing method, wherein the above low-resolution fundus image and the high-resolution fundus image are taken of the same patient, wherein the low-resolution fundus image is a fundus image of the patient before eye disease surgery, and the high-resolution fundus image is a fundus image after eye disease surgery.

12. In paragraph 10, An image processing method, wherein the learning data includes data augmented through preprocessing that rotates each image by a predetermined angle, preprocessing that performs linear interpolation on low-quality and high-quality images, preprocessing that performs blur or haze processing on each image, or preprocessing that adjusts the brightness of each image.

13. A non-transitory computer-readable recording medium having recorded thereon a computer program for performing an image processing method according to any one of claims 7 to 12.

Images