COMPUTER-IMPLEMENTED IMAGE PREPROCESSING FOR A FUNDOSCOPICAL IMAGE

Description

BACKGROUNDTechnical Field

[0001] The disclosure relates to a computer-implemented image processing technology, and particularly relates to an image pre-processing method and an image processing apparatus for a fundoscopic image.Description of Related Art

[0002] Medical images are images taken of specific parts of organisms, and these images may be used to assess the risk or severity of a disease. For example, through fundoscopic photographic examinations, diseases such as retinopathy, glaucoma, macular disease or the like may be detected early. Generally speaking, most doctors judge lesions in the medical images manually. Although computer-assisted evaluation of medical images is now possible, breakthroughs are needed in terms of indicators such as efficiency, complexity, and accuracy.

[0003] US 2015 / 110368 A1 discloses systems and methods that aid in screening, diagnosis and / or monitoring of medical conditions. The systems and methods may allow, for example, for automated identification and localization of lesions and other anatomical structures from medical data obtained from medical imaging devices, computation of image-based biomarkers including quantification of dynamics of lesions, and / or integration with telemedicine services, programs, or software.

[0004] The article "Automatic Detection of Microaneurysms in Color Fundus Images of the Hunan Retina by Means of the Bounding Box Closing" by Walter Thomas et al., published on Oct. 10, 2002 in "Springer International Publishing", XP055892492, vol. 2526, pages 210-220, discloses a new algorithm for the detection of microaneurysms in color fundus images of the human retina. Microaneurysms are the first unequivocal indication of Diabetic Retinopathy (DR), a severe and wide-spread eye disease. Their automatic detection may play a major role in computer assisted diagnosis of DR. We propose an algorithm that can be divided into four steps. The first step is an image enhancement technique that comprises normalization and noise reduction. The second step is the extraction of small details that fulfill a certain criterion: This leads to the definition of the bounding box closing. Then, an automatic threshold depending on image quality is calculated. In the last step false positives are eliminated.

[0005] WO 2019 / 013779A1 discloses a method for vessel mapping and quantification. The method comprises pre-processing a retinal image to generate a vessel segmented image and processing the vessel segmented image to generate an image with a central light reflex. The method further includes identifying a cylindrical region in the central light reflex. The image is further processed to apply a morphological skeletonization operation to generate vessel centerlines and the segmented vascular network of the retinal image.

[0006] US 2007 / 248277 A1 discloses methods and system for processing image data including receiving image data corresponding to an image, where the image data describes a number of pixels. A blurring filter is applied to the image data, where the blurring filter reduces contrast between at least some of the pixels. A stretching filter is applied to the image data, where the stretching filter increases the number of pixels. A sharpening filter is applied to the image data, where the sharpening filter increases contrast between at least some of the pixels.

[0007] US 2005 / 169553 A1 discloses an image sharpening which is performed by applying variable contrast stretching to pixels of interest in a digital image. For each pixel of interest, the amount of contrast stretching is a function of minimum and maximum intensity values in a local pixel neighborhood.

[0008] The article "A review on automatic analysis techniques for color fundus photographs" by Bescenczi Renátó et al., published on Oct. 6, 2016 in "Computational and Structural Biotechnology Journal", vol. 14, pages 371-384, XP055891577, discloses a review on automatic image processing tools to recognize diseases causing specific distortions in the human retina. After a brief summary of the biology of the retina, we give an overview of the types of lesions that may appear as biomarkers of both eye and non-eye diseases. We present several state-of-the-art procedures to extract the anatomic components and lesions in color fundus photographs and decision support methods to help clinical diagnosis. We list publicly available databases and appropriate measurement techniques to compare quantitatively the performance of these approaches. Furthermore, we discuss on how the performance of image processing-based systems can be improved by fusing the output of individual detector algorithms. Retinal image analysis using mobile phones is also addressed as an expected future trend in this field.SUMMARY

[0009] The present invention is defined by the appended claims. Embodiments of the disclosure provide a computer-implemented image pre-processing method and an image processing apparatus for a fundoscopic image, in which features can be enhanced, thereby improving accuracy of subsequent identification of lesions or other features.

[0010] The computer-implemented image pre-processing method according to claim 1 includes the following steps. A region of interest is obtained from a fundoscopic image to generate a first image comprising only the region of interest as information to be used in subsequent image processing. The region of interest is focused on and corresponds to an eyeball in the fundoscopic image. A smoothing process is performed on the first image to generate a second image. For each pixel a distance between a pixel value difference and a reference value is determined, wherein the pixel value difference is a difference between a value of said pixel and a value of a neighboring pixel in the second image. And updating is performed by increasing, for each pixel, the pixel value difference according to the determined distance. A third image is generated in which the values of the pixels of said second image are changed such that, for each pixel, the pixel value difference between said pixel and the neighboring pixel matches the updated pixel value difference. The third image is used for image recognition.

[0011] The image processing apparatus according to claim 8 includes a storage and a processor. The storage stores a code. The processor is coupled to the storage. The processor loads and executes the code to be configured to: obtain a region of interest from a fundoscopic image to generate a first image comprising only the region of interest as information to be used in subsequent image processing; perform a smoothing process on the first image to generate a second image; determine, for each pixel, a distance between a pixel value difference and a reference value, wherein the pixel value difference is a difference between a value of said pixel and a value of a neighboring pixel in the second image; and update by increase, for each pixel, the pixel value difference according to the determined distance. A third image is generated in which the values of the pixels of said second image are changed such that, for each pixel, the pixel value difference between said pixel and the neighboring pixel matches the updated pixel value difference. The region of interest is focused on and corresponds to an eyeball in the fundoscopic image. The third image is used for image recognition.

[0012] Based on the above, according to the image pre-processing method and the image processing apparatus for a fundoscopic image according to the embodiments of the disclosure, before image recognition is performed, the region of interest is cut out from an initial fundoscopic image, and the smoothing process and value enhancement are further performed. In this way, features may be enhanced, noises may be reduced, and accuracy of the subsequent image recognition may be improved.

[0013] To make the aforementioned more comprehensible, several embodiments accompanied with drawings are described in detail as follows.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are included to provide a further understanding of the disclosure, and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the principles of the disclosure. FIG. 1 is a block diagram of components of an image processing apparatus according to an embodiment of the disclosure. FIG. 2 is a flowchart of an image pre-processing method according to an embodiment of the disclosure. FIG. 3 is a schematic diagram of a first image according to an embodiment of the disclosure. FIG. 4 is a schematic diagram of a second image according to an embodiment of the disclosure. FIG. 5 is a schematic diagram of a third image according to an embodiment of the disclosure. DESCRIPTION OF THE EMBODIMENTS

[0015] FIG. 1 is a block diagram of components of an image processing apparatus 100 according to an embodiment of the disclosure. Referring to FIG. 1, the image processing apparatus 100 includes (but is not limited to) a storage 110 and a processor 130. The image processing apparatus 100 may be a desktop computer, a notebook computer, a smart phone, a tablet computer, a server, a medical testing instrument, or other computing apparatuses.

[0016] The storage 110 may be any type of fixed or removable random access memory (RAM), read only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD) or similar components. In one embodiment, the storage 110 is configured to record a code, a software module, a configuration, data (for example, an image, a value, a reference value, a distance, etc.) or a file, and the embodiments thereof will be described in detail later.

[0017] The processor 130 is coupled to the storage 110. The processor 130 may be a central processing unit (CPU), a graphic processing unit (GPU), or other programmable general-purpose or special-purpose microprocessor, digital signal processor (DSP), programmable controller, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), neural network accelerator or other similar components or a combination of the above components. In one embodiment, the processor 130 is configured to execute all or part of operations of the image processing apparatus 100, and may load and execute each code, the software module, the file, and the data recorded by the storage 110.

[0018] Hereinafter, the method according to the embodiments of the disclosure will be described with reference to each apparatus, component, and module in the image processing apparatus 100. Each process of the method may be adjusted according to actual implementation, and is not limited to those described herein.

[0019] FIG. 2 is a flowchart of an image pre-processing method according to an embodiment of the disclosure. Referring to FIG. 2, the processor 130 may obtain a region of interest from a fundoscopic image to generate a first image (step S210). Specifically, the fundoscopic image is an image obtained by fundoscopic photography of a human or other organism. The processor 130 may obtain the fundoscopic image through a built-in or external image capturing apparatus, or the processor 130 may download the fundoscopic image from a server, a computer, or a storage medium. It should be noted that fundoscopic images from different sources may have different shapes or sizes. In order to normalize these fundoscopic images, the processor 130 may first cut out the region of interest considered as important or useful information.

[0020] The region of interest according to the embodiments of the disclosure is focused on an eyeball in the fundoscopic image. In one embodiment, the processor 130 may locate a center of the eyeball from the fundoscopic image. For example, the processor 130 may take a point where the most straight lines in a gradient direction of pixels intersect in the fundoscopic image as a position of the eyeball. For another example, the processor 130 may perform the Hough transformation on the fundoscopic image to select a circle that best meets the requirements, and determine a center of circle of a contour of the region of interest accordingly. The processor 130 may further determine the region of interest according to the center of the eyeball. For example, the processor 130 may set a circle that relatively or most conforms to the contour of the eyeball according to the center, and use the contour of the circle as a boundary of the region of interest. For another example, a circle obtained by the Hough transformation may be used as the region of interest by the processor 130.

[0021] In another embodiment, the processor 130 may search for a boundary of the eyeball from the outside to the center of the fundoscopic image. For example, the processor 130 scans the fundoscopic image sequentially from four sides toward the center, and determines lightness of a scanned region. It should be noted that a lightness value (or brightness, that is, the lightness of a color) on one side of the boundary of the eyeball is higher than the other side of the boundary of the eyeball. Generally speaking, a region outside the eyeball in the fundoscopic image has a low lightness value and may be in black. When a lightness difference between neighboring pixels on any side is higher than a difference threshold, or the lightness value of one or more pixels is higher than a lightness threshold (that is, the lightness value on one side is higher than that on the other side), the processor 130 may determine that an outermost edge of the region of interest on this side has been found. The processor 130 may use the outermost edges respectively found on the four sides of the fundoscopic image as boundary lines. That is, a quadrilateral is formed. The processor 130 may use a length of a shortest side of the quadrilateral as a diameter of a circle (taken as the eyeball), and use the circle formed by the diameter as the boundary of the eyeball. The center of the circle is the center of the quadrilateral. In another embodiment, in order to first filter out interference that often appears on a periphery, the processor 130 multiplies half the length of the shortest side of the quadrilateral by a floating point number greater than 0 and less than 1 to obtain a radius length, and uses the radius length to obtain a circle. In addition, the center of the circle is still at the center of the quadrilateral. Then, the processor 130 may determine the region of interest according to the boundary of the eyeball. That is, the boundary of the eyeball is taken as the boundary of the region of interest.

[0022] In one embodiment, the processor 130 may cut out the region of interest from the fundoscopic image. That is, the processor 130 may delete a region in the fundoscopic image that is not the region of interest. The processor 130 may further add a background color outside the region of interest to form the first image. This background color will be considered as useless information in subsequent image recognition (for example, for lesion identification or for severity identification) of the fundoscopic image. The useless information may be excluded by feature extraction or may have a relatively low value. For example, red, green and blue background colors all consist of values of 128, 64, or 0, but are not limited thereto. In addition, the size, shape, and / or ratio of the first image may be fixed, thereby normalizing different fundoscopic images. In some embodiments, the circle may be changed to an ellipse or other geometric figures.

[0023] For example, FIG. 3 is a schematic diagram of the first image according to an embodiment of the disclosure. Referring to FIG. 3, a circular region (i.e., the region of interest) shown in the figure corresponds to the eyeball.

[0024] The processor 130 may perform a smoothing process on the first image to generate a second image (step S230). Specifically, the smoothing process is a spatial domain filtering technology capable of directly blurring a pixel in an image and removing noises. For example, a value difference (also referred to as distance) between neighboring pixels may be reduced.

[0025] In one embodiment, the smoothing process is Gaussian blur. The processor 130 may perform Gaussian blur on the first image. For example, the processor 130 performs a convolution operation on each pixel in the first image by using a Gaussian convolution kernel, and then sums convolution results to obtain the second image.

[0026] For example, FIG. 4 is a schematic diagram of the second image according to an embodiment of the disclosure. Referring to FIG. 3 and FIG. 4, compared with FIG. 3, FIG. 4 shows that details of some noises are blurred by Gaussian blur, while edges of blood vessels, macula, veins and / or arteries remain visible.

[0027] In other embodiments, the smoothing processing may also be median filtering, mean filtering, box filtering or other processing.

[0028] The processor 130 may increase a value difference between multiple neighboring pixels in the second image to generate a third image (step S250). Specifically, the smoothing process may narrow the value difference between the neighboring pixels. In order to further enhance a feature, in one embodiment, the processor 130 may increase (i.e., update or change) the value difference in proportion to a distance between the value difference between the neighboring pixels and a reference value. For example, when the reference value is 128, the processor 130 may calculate an original value difference of red, green, and blue channel values between each pixel and its neighboring pixel, and compare the distances between the original value differences and the reference value. As the distance is increased, the processor 130 may increase the magnitude of an increase in the value difference. As the distance is reduced, the processor 130 may reduce the magnitude of an increase in the value difference. The proportion may be 1, 2, 5, or 10 times. Then, according to an increased value difference (i.e., updated value difference), the processor 130 may change the values of the corresponding pixels so that the value difference between two pixels matches the updated value difference.

[0029] In some embodiments, a value to be changed has an upper limit or lower limit. For example, the upper limit may be 255 and the lower limit may be 0. When the changed value exceeds the upper limit or the lower limit, it may be set as a specific value (for example, the upper limit, the lower limit or other values).

[0030] It should be noted that a mathematical relationship between the original value difference and the updated value difference is not limited to a proportional relationship. In other embodiments, the processor 130 may also adopt a linear relationship, an exponential relationship, or other mathematical relationships depending on actual needs.

[0031] For example, FIG. 5 is a schematic diagram of the third image according to an embodiment of the disclosure. Referring to FIG. 4 and FIG. 5, the blood vessels, the macula, the optic disc, the veins, the arteries or other objects in FIG. 5 are more clear.

[0032] It should also be noted that the third image according to the embodiments of the disclosure may be used for image recognition. In one embodiment, the processor 130 inputs the third image to a detection model based on a machine learning algorithm (for example, deep neural network (DNN), multi-layer perceptron (MLP), support vector machine (SVM) or other machine learning models). In one embodiment, the detection model may be used for image recognition. It should be noted that the third image may be used in pre-processing in a training phase and / or inference phase of the detection model. Generally speaking, in the detection model, an initial fundoscopic image is usually immediately subjected to feature extraction without undergoing image pre-processing. Through the image pre-processing according to the embodiments of the disclosure, a relatively accurate recognition result of an image of a lesion such as bleeding, exudates, and edema can be obtained. Alternatively, the image pre-processing according to the embodiments of the disclosure may facilitate identification of a part such as blood vessels, macula, or veins, but the disclosure is not limited to the above.

[0033] In another embodiment, the image recognition may be based on scale-invariant feature transform (SIFT), a Haar-like feature, AdaBoost, or other recognition technologies.

[0034] To sum up, in the image pre-processing method and the image processing apparatus for a fundoscopic image according to the embodiments of the disclosure, the region of interest in the fundoscopic image is determined, the smoothing process is performed on the image, and the value difference is increased. In this way, features may be enhanced, and the subsequent image recognition, model training, or other image applications may be facilitated.

Claims

1. A computer-implemented image pre-processing method, comprising: obtaining a region of interest from a fundoscopic image to generate a first image comprising only the region of interest as information to be used in subsequent image processing, wherein the region of interest is focused on and corresponds to an eyeball in the fundoscopic image; performing a smoothing process on the first image to generate a second image; determining, for each pixel, a distance between a pixel value difference and a reference value, wherein the pixel value difference is a difference between a value of said pixel and a value of a neighboring pixel in the second image; and updating by increasing, for each pixel, the pixel value difference according to the determined distance; generating a third image in which the values of the pixels of said second image are changed such that, for each pixel, the pixel value difference between said pixel and the neighboring pixel matches the updated pixel value difference.

2. The computer-implemented image pre-processing method according to claim 1, wherein obtaining the region of interest from the fundoscopic image comprises: locating a center of the eyeball from the fundoscopic image; and determining the region of interest according to the center.

3. The computer-implemented image pre-processing method according to claim 1, wherein obtaining the region of interest from the fundoscopic image comprises: finding a boundary of the eyeball from an outside of the fundoscopic image, wherein a lightness value on one side of the boundary of the eyeball is higher than the lightness value on the other side of the boundary of the eyeball; and determining the region of interest according to the boundary of the eyeball.

4. The computer-implemented image pre-processing method according to claim 1, wherein generating the first image comprises: cutting out the region of interest from the fundoscopic image; and adding a background color outside the region of interest to form the first image.

5. The computer-implemented image pre-processing method according to claim 1, wherein the smoothing process is Gaussian blur, and performing the smoothing process on the first image comprises: performing the Gaussian blur on the first image.

6. The computer-implemented image pre-processing method according to claim 1, wherein updating by increasing the pixel value difference according to the determined distance comprises: increasing the pixel value difference in proportion to the distance between the pixel value difference and the reference value.

7. The computer-implemented image pre-processing method according to claim 1, further comprising, after generating the third image: inputting the third image to a detection model based on a machine learning algorithm.

8. An image processing apparatus (100), comprising: a storage (110), storing a code; and a processor (130), coupled to the storage (110), loading and executing the code to be configured to: obtain a region of interest from a fundoscopic image to generate a first image comprising only the region of interest as information to be used in subsequent image processing, wherein the region of interest is focused on and corresponds to an eyeball in the fundoscopic image; perform a smoothing process on the first image to generate a second image; determine, for each pixel, a distance between a pixel value difference and a reference value, wherein the pixel value difference is a difference between a value of said pixel and a value of a neighboring pixel in the second image; and update by increasing, for each pixel, the pixel value difference according to the determined distance; generate a third image in which the values of the pixels of said second image are changed such that, for each pixel, the pixel value difference between said pixel and the neighboring pixel matches the updated pixel value difference.

9. The image processing apparatus (100) according to claim 8, wherein the processor (130) is further configured to: locate a center of the eyeball from the fundoscopic image; and determine the region of interest according to the center.

10. The image processing apparatus (100) according to claim 8, wherein the processor (130) is further configured to: find a boundary of the eyeball from an outside to a center of the fundoscopic image, wherein a lightness value on one side of the boundary of the eyeball is higher than the lightness value on the other side of the boundary of the eyeball; and determine the region of interest according to the boundary of the eyeball.

11. The image processing apparatus (100) according to claim 8, wherein the processor (130) is further configured to: cut out the region of interest from the fundoscopic image; and add a background color outside the region of interest to form the first image.

12. The image processing apparatus (100) according to claim 8, wherein the smoothing process is Gaussian blur, and the processor (130) is further configured to: perform the Gaussian blur on the first image.

13. The image processing apparatus (100) according to claim 8, wherein the processor (130) is further configured to: increase the pixel value difference in proportion to the distance between the pixel value difference and the reference value.

14. The image processing apparatus (100) according to claim 8, wherein the processor (130) is further configured to: input the third image to a detection model based on a machine learning algorithm.