Method for detecting ocular hyperemia and apparatus therefor

Abstract

The present disclosure relates to a technique for detecting the presence of hyperemia in the eye and provides an apparatus and method for detecting ocular hyperemia, the apparatus comprising: an image acquisition unit that acquires image information; a feature point acquisition unit that acquires eye region feature points of an eye region image extracted from the image information; an eye sub-region extraction unit that extracts an eye sub-region image through segmentation using the eye region feature points; a conjunctival region image generation unit that detects an iris region from the eye region image or the eye sub-region image and generates a conjunctival region image by using the eye sub-region image and the iris region; and a determination unit that inputs the conjunctival region image to a pre-trained hyperemia determination algorithm, so as to derive a classification result value regarding the presence or absence of hyperemia.

Description

Method for detecting ocular congestion and device therefor

[0001] The present disclosure relates to a technique for detecting whether an eye is congested.

[0002] Bloodshot eyes can be a symptom of various diseases, and even in normal situations, they can be caused by a variety of factors. However, the most accurate way to diagnose bloodshot eyes is to visit an ophthalmologist with a specialist and use medical equipment. However, this requires a hospital visit and examination, which is time-consuming and costly.

[0003] To avoid this inconvenience, prior art has provided similar techniques for assessing diabetic macular edema, using specialized medical cameras to capture images of specific areas of the eye for hemorrhage. However, these techniques also require specialized medical cameras, which pose limitations in terms of cost and widespread adoption.

[0004] When assessing congestion using a standard camera, the image is used to calculate probability values ​​for each pixel, thereby probabilistically determining whether or not the eye is congested. In particular, using pixel probability values ​​reduces the accuracy of eye region detection, and noise from areas other than the eye significantly impacts the judgment.

[0005] To solve these problems, we aim to provide a technology that can accurately and easily determine whether the eye is bloodshot using an RGB camera.

[0006] The present disclosure seeks to provide a technique for detecting whether the eye is congested.

[0007] In one aspect, the present embodiments provide an ocular congestion detection device including an image acquisition unit that acquires image information, a feature acquisition unit that acquires eye region feature points of an eye region image extracted from the image information, an eye detail region extraction unit that segments and extracts an eye detail region image using the eye region feature points, a conjunctival region image generation unit that detects an iris region from an eye region image or an eye detail region image and generates a conjunctival region image using the eye detail region image and the iris region, and a judgment unit that inputs the conjunctival region image into a pre-learned congestion judgment algorithm to derive a classification result value for whether or not there is congestion.

[0008] In another aspect, the present embodiments provide an ocular congestion detection method for detecting whether or not the eye is congested, the method including an image acquisition step of acquiring image information, a feature acquisition step of acquiring eye region feature points of an eye region image extracted from the image information, an eye region extraction step of segmenting and extracting an eye region image using the eye region feature points, a conjunctival region image generation step of detecting an iris region from the eye region image or the eye region image and generating a conjunctival region image using the eye region image and the iris region, and a judgment step of inputting the conjunctival region image into a pre-learned congestion judgment algorithm to derive a classification result value for whether or not the eye is congested.

[0009] According to the present disclosure, a technique for detecting whether an eye is congested can be provided.

[0010] FIG. 1 is a diagram illustrating a configuration of an ocular congestion detection device according to one embodiment.

[0011] FIG. 2 is a diagram for explaining an operation of processing image information according to one embodiment.

[0012] FIG. 3 is a diagram for explaining an operation of segmenting an eye sub-region image according to one embodiment.

[0013] FIG. 4 is a drawing for explaining an operation of detecting an iris region according to one embodiment.

[0014] FIG. 5 is a drawing for explaining an operation of detecting an iris area according to another embodiment.

[0015] FIG. 6 is a diagram for explaining an operation of generating a conjunctival area image according to one embodiment.

[0016] FIG. 7 is a diagram for explaining a data set of a charge judgment algorithm according to one embodiment.

[0017] Fig. 8 is a flowchart for explaining a method for detecting ocular congestion according to one embodiment.

[0018] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When adding reference numerals to components in each drawing, identical components may have the same numerals as much as possible even if they are shown in different drawings. In addition, when describing the present embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the technical idea of ​​the present invention, the detailed description may be omitted. When "includes," "has," "consists of," etc. are used in this specification, other parts may be added unless "only" is used. When a component is expressed in the singular, it may include a case in which the plural is included unless specifically stated otherwise.

[0019] Additionally, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the present disclosure. These terms are only intended to distinguish the components from other components, and the nature, order, sequence, or number of the components are not limited by the terms.

[0020] In a description of the positional relationship of components, when it is described that two or more components are "connected," "combined," or "connected," it should be understood that the two or more components may be directly "connected," "combined," or "connected," but that the two or more components may also be further "interposed" with another component to be "connected," "combined," or "connected." Here, the other component may be included in one or more of the two or more components that are "connected," "combined," or "connected" to each other.

[0021] In the description of the temporal flow relationship related to components, operation methods, or manufacturing methods, for example, when the temporal or flow relationship is described as “after”, “following”, “next to”, “before”, etc., it may also include cases where it is not continuous, unless “immediately” or “directly” is used.

[0022] Meanwhile, when numerical values ​​or corresponding information (e.g., levels, etc.) for components are mentioned, even without separate explicit description, the numerical values ​​or corresponding information may be interpreted as including an error range that may occur due to various factors (e.g., process factors, internal or external impact, noise, etc.).

[0023] The embodiments are described in detail with reference to the drawings below.

[0024] Eye congestion can be a key indicator of various health problems. While there are various medical causes of eye congestion, its occurrence can lead to health problems and impaired ocular function, potentially leading to dangerous situations.

[0025] In particular, industrial sites are prone to numerous environmental hazards, such as dust and sand, and the risk of workers experiencing bloodshot eyes is high. Furthermore, areas where bloodshot eyes can occur, such as working at heights or in hazardous areas, are relatively widespread.

[0026] Therefore, there is a need to easily screen industrial workers for bloodshot eyes before they begin work. This will help ensure a safe working environment and prevent accidents.

[0027] However, in the past, it was possible to determine whether there was congestion based on pixel probability values ​​by directly capturing the cornea using a medical camera or capturing an image using a general camera.

[0028] These conventional technologies require expensive equipment and are sensitive to external conditions, making them difficult to install and operate in industrial settings. Furthermore, when assessing congestion using pixel color or probability values ​​in images, the limitations of standard cameras make it difficult to precisely extract only the cornea and assess congestion, reducing reliability.

[0029] Accordingly, the present disclosure aims to provide a technology capable of quickly and accurately determining whether or not the eye is congested by using an image simply captured using an RGB camera such as a mobile phone camera in an industrial setting, etc.

[0030] For example, the present embodiments provide a technology that allows a user to capture images using a mobile phone camera or a standard camera installed in an industrial setting, and allows an ocular congestion detection device to quickly extract only the corneal region from the image information and use a preset artificial intelligence algorithm to determine a classification value for congestion. This technology can provide highly accurate judgment results while being highly applicable to various industrial settings.

[0031] Below, the devices and methods according to the present disclosure are described in more detail with reference to the drawings. The term "eyeball" below may refer to an eye, and the term "iris" may refer to a portion including the pupil. Furthermore, "feature points" may refer to landmarks, and each algorithm described below is merely exemplary, and various algorithms that perform the same purpose and function may be applied to the present disclosure. Furthermore, programs capable of performing the functions of the present embodiments are also included in the present disclosure, and a recording medium containing the programs is also construed as being included in the present disclosure.

[0032] FIG. 1 is a diagram illustrating a configuration of an ocular congestion detection device according to one embodiment.

[0033] Referring to FIG. 1, the ocular congestion detection device (100) may include an image acquisition unit (110) that acquires image information, a feature acquisition unit (120) that acquires eye region feature points of an eye region image extracted from the image information, an eye region extraction unit (130) that segments and extracts an eye region image using the eye region feature points, a conjunctival region image generation unit (140) that detects an iris region from an eye region image or an eye region image and generates a conjunctival region image using the eye region image and the iris region, and a judgment unit (150) that inputs a conjunctival region image into a pre-learned congestion judgment algorithm to derive a classification result value for whether or not there is congestion.

[0034] For example, the image acquisition unit (110) can receive image information captured from a sensor such as a camera. Here, the camera refers to a general camera such as the RGB camera described above. Alternatively, the image acquisition unit (110) can also acquire image information by capturing frames from an image.

[0035] The feature point acquisition unit (120) can acquire an eye region image including an eye from the acquired image information. For example, the eye region image can be set to include eye region feature points but exclude other region feature points included in the facial image information. For example, the image information can include a background, a person's facial image information, and other body images such as a neck. Once the image information is acquired, the feature point acquisition unit (120) can extract the person's facial image information from the image information using an artificial intelligence algorithm. For this purpose, a landmark extractor that acquires the facial image information from the image information can be used. The landmark extractor performs a function of extracting feature points related to the outline of the person's face from the image, and in the following description, landmarks are described as feature points.

[0036] Once facial image information is extracted, facial image information including the eye region can be identified. As described above, the facial image information can be configured to include only feature points assigned to the eye region. Through this, the feature point acquisition unit (120) can acquire eye region feature points from the image information.

[0037] The eye sub-region extraction unit (130) can extract an eye sub-region image using eye sub-region feature points and a preset interpolation algorithm. The eye sub-region image includes the eye sub-region feature points, but may also include images of the skin around the eye and eyebrows, etc., in addition to the eye. Therefore, in order to accurately determine whether there is congestion, the eye sub-region extraction unit (130) must extract only the eye sub-region image from the eye sub-region image.

[0038] To this end, the eye sub-region extraction unit (130) can utilize eye sub-region feature points. The eye sub-region extraction unit (130) can determine the outline of the eye sub-region using the acquired eye sub-region feature points and an image processing algorithm. Once the outline is determined, the eye sub-region extraction unit (130) can extract only the portion within the outline as an eye sub-region image.

[0039] For example, a preset interpolation algorithm may be set to extract an eye sub-region image by predicting the outer coordinate values ​​of the eye sub-region based on the two-dimensional coordinate values ​​of the eye region feature points. For example, when the two-dimensional coordinate values ​​for N eye region feature points are confirmed, the interpolation algorithm may predict the coordinate values ​​between the eye region feature points using the two-dimensional coordinate values. For example, various algorithms such as Forward / Backward Interpolation, Linear Interpolation, Bilinear Interpolation, and Polynomial Interpolation may be used as the interpolation algorithm. Through this, the eye sub-region extraction unit (130) may obtain the outer coordinate values ​​connecting the eye region feature points and extract the outline of the eye region by connecting them.

[0040] When the eye detail region extraction unit (130) extracts an outline, it can extract an image based on this and generate an image only for the eye detail region within the outline.

[0041] The conjunctival region image generation unit (140) can generate a conjunctival region image by isolating only the conjunctival region that serves as a criterion for determining whether or not there is congestion. To this end, the conjunctival region image generation unit (140) must detect the iris region from the eye's detailed region.

[0042] For example, the conjunctival region image generation unit (140) may calculate the sum of gray level values ​​for pixels around a circle set to arbitrary center coordinates and radius values ​​on the eye region image or the eye sub-region image, and may set the circumference of the circle where the change in the sum of gray level values ​​is maximum according to the change in the radius value as the boundary of the iris region. For example, the conjunctival region image generation unit (140) may set arbitrary coordinates within the eye region image or the eye sub-region image as the center coordinates, may also set the radius value to an arbitrary value, and may calculate the sum of gray level values ​​for pixels around the circle for each arbitrary center coordinate and each arbitrary radius value.

[0043] The conjunctival region image generation unit (140) can continuously calculate the sum of these gray level values ​​while changing the center coordinate and radius value. The conjunctival region image generation unit (140) sets the circumference of the circle where the change in the sum of the calculated gray level values ​​is the largest as the boundary of the iris region, and stores the center coordinate and radius value at this time. For example, the conjunctival region image generation unit (140) can determine the center coordinate using an experimental range for the center coordinate of the eyeball during the process of generating an eyeball region image or an eyeball sub-region image. That is, in the eyeball region image or eyeball sub-region image, the center point of the iris (i.e., the center point of the eyeball) is formed in the central part of the image. Therefore, for fast calculation processing, the conjunctival region image generation unit (140) preferentially sets the center coordinate to an arbitrary value within a preset coordinate range. Similarly, the radius value can also be set to an arbitrary range so that it is first selected within a certain range depending on the image scale, etc. Through this action, the conjunctival region image generation unit (140) can detect the iris region more quickly.

[0044] As another example, the conjunctival region image generation unit (140) can determine a certain number of candidate iris regions using a circle detection algorithm, determine the pupil center point using a DeepEye model, and select the iris region corresponding to the pupil center point from the candidate iris regions to determine the final iris region. Through this, in cases where multiple candidate iris regions appear through the circle detection algorithm, such as eyelashes, a more accurate iris region can be detected. The circle detection algorithm can be an algorithm that detects by utilizing changes in pixels while changing the aforementioned center coordinates and radius values.

[0045] As another example, the conjunctival region image generation unit (140) can detect the iris region by applying a convex hull algorithm to the eye region image or the eye sub-region image. For example, the conjunctival region image generation unit (140) can detect the iris region by using a convex hull algorithm that connects the outermost parts of a plurality of data values ​​and represents them in a polygonal shape using the eye sub-region image. The shape of the iris may not appear circular due to the eyelids of the human body, etc. Therefore, the iris region detection using the center coordinates and radius values ​​described above may incur errors depending on the situation. In particular, when the iris region is detected using the eye sub-region image instead of the eye region image, there is a high possibility that the iris will not appear circular. Therefore, the conjunctival region image generation unit (140) can more accurately acquire the shape of the iris region by using an image processing algorithm such as a convex hull. For example, the conjunctival region image generation unit (140) can represent the outermost part of the given data values ​​(so that all the given data values ​​are included) on a two-dimensional plane of the eye detailed region image in the form of a polygon and detect the interior of the polygon as the iris region.

[0046] In addition to the above-described operation, the conjunctival region image generation unit (140) can detect the iris region using an image processing algorithm or artificial intelligence algorithm capable of detecting the iris region.

[0047] When the iris region is detected, the conjunctival region image generation unit (140) can generate a conjunctival region image only for the remaining region excluding the iris region from the eye detailed region image. That is, the conjunctival region image generation unit (140) can generate a conjunctival region image so that only the conjunctival region remains by masking the iris region from the eye detailed region image that includes only the conjunctiva and iris regions.

[0048] For example, the conjunctival region image generation unit (140) can generate a conjunctival region image by performing bit operations on the iris region and eye sub-region images. For example, the conjunctival region image generation unit (140) can generate a conjunctival region image so that only the conjunctival region appears by performing bit operations on the detected iris region and eye sub-region images. That is, the conjunctival region image generation unit (140) can generate a conjunctival region image so that only the conjunctival region remains by making the overlapping area of ​​the eye sub-region image and the iris region 0 through image bit operations.

[0049] As another example, the conjunctival region image generation unit (140) can generate a conjunctival region image by zero-masking the iris region in the eye detailed region image so that only the conjunctival region remains.

[0050] Through the aforementioned operation, the ocular congestion detection device (100) can preprocess image information to generate a conjunctival region image. This enables a highly accurate determination of ocular congestion.

[0051] The ocular congestion detection device (100) may include a judgment unit (150) that inputs a conjunctival region image into a pre-learned congestion judgment algorithm to derive a classification result value for congestion. The ocular congestion detection device (100) may input a conjunctival region image into a pre-learned congestion judgment algorithm that has been trained to determine whether congestion is present.

[0052] For example, the pre-learned congestion judgment algorithm is an artificial intelligence algorithm learned using the entire data set including congested conjunctiva area image data and normal conjunctiva area image data.

[0053] For example, the pre-trained congestion detection algorithm is an algorithm trained using a CNN-based artificial intelligence algorithm. To improve the accuracy of congestion detection, a conjunctival region image can be extracted from the aforementioned image information to build a dataset for the congestion detection algorithm. For example, once the image information is acquired, the congestion detection algorithm learner performs a preprocessing step in which it extracts an eye sub-region image using feature points from the eye region image, and then extracts a conjunctival region image using the iris region and eye sub-region images.

[0054] Afterwards, when the conjunctival region image is extracted, the learning machine learns the congestion judgment algorithm using the entire data set including conjunctival region image data and normal conjunctival region image data.

[0055] Additionally, the entire dataset is divided into a training dataset used during the learning process and a validation dataset used to calculate accuracy while changing at least one of the epoch, hyperparameter, and hidden layer. The validation dataset is used during the learning process but not in the actual learning itself. It is used to verify the learning using the training dataset. Because it is used during the learning process, it is composed of a different dataset from a typical test set.

[0056] While the learner is learning using the training data set, the optimal epoch that avoids overfitting can be identified by increasing the number of epochs and calculating the accuracy of the validation data set. Similarly, while learning using the training data set, the optimal number of parameters and layers can be determined by varying the number of hyperparameters or hidden layers and calculating the accuracy of the validation data set.

[0057] Additionally, in order to verify the validity of the data set during the learning process, an operation may be performed to verify the validity of using the preprocessed conjunctival region image by applying the image information in RGB format before preprocessing to an algorithm such as Grad-CAM, which is an explainable artificial intelligence.

[0058] The judgment unit (150) inputs a conjunctival region image as input to a pre-learned congestion judgment algorithm learned through the aforementioned process, obtains a classification value for congestion as output, and determines whether the eye is congested in the image.

[0059] Through the above steps, it is possible to accurately determine whether or not there is congestion even using images captured using a standard RGB camera sensor. Furthermore, by appropriately combining AI and image processing algorithms, the increased processing time that can occur when using AI alone can be prevented.

[0060] Below, the congestion judgment operation according to the aforementioned ocular congestion detection device is described in more detail and in various ways with reference to drawings.

[0061] FIG. 2 is a diagram for explaining an operation of processing image information according to one embodiment.

[0062] Referring to Fig. 2, the image acquisition unit can receive image information (200) captured by an RGB camera. The image information (200) can include various objects, including the face of the subject of the capture. For example, the image information (200) can include background information. The background information can be varied in various ways when used in industrial settings, etc., and since it is not captured in a fixed location and under fixed conditions, as with a medical camera, there is a high possibility of errors occurring.

[0063] Therefore, a preprocessing step is required to extract an image for a necessary portion from the image information (200). To this end, the feature point acquisition unit can acquire an eye region image (220, 225) from the image information (200). The eye region image includes feature points of the eye region and can be set to exclude feature points occurring in other regions.

[0064] For example, when image information (200) is acquired, the feature point acquisition unit can detect a facial region (210). For example, the facial region (210) can be configured with a deep learning-based algorithm and detected using a Convolutional Neural Network (CNN). Many algorithms for detecting the facial region (210) are publicly available, and there are no limitations as long as the facial region (210) can be detected.

[0065] However, in this embodiment, it is important to acquire feature points for the eye area (220, 225) in the facial area (210). Therefore, the feature point acquisition unit acquires feature points for the eye area (220, 225). For example, landmarks of the face can be found by using a CNN based on Cascade, which is an ensemble learning method that combines multiple hierarchical structures or classifiers, or by using Auto-Encoder-based learning that finds hidden patterns in data. For example, APIs such as Mediapipe, Openpose, MMPose, and OpenFace can be used. In addition, feature points for the eye area (220, 225) can be extracted through various image processing techniques, and there are no limitations thereon.

[0066] The facial region (210) can be detected through feature point extraction for image information (200). The feature point can be called a landmark and can correspond to the coordinate values ​​for a characteristic part such as color or boundary on the image.

[0067] Meanwhile, when the feature point acquisition unit secures the feature points for the eye region (220, 225), the two eye regions (220, 225) can be separated to perform the operation of the ocular congestion detection device described above. The eye regions (220, 225) can be divided into two, left and right, and for each image, eye detailed region image extraction, conjunctival region image generation, and classification result value derivation can be performed. However, in order to establish that they are the same user, the separated left and right eye regions (220, 225) can be mapped to the same flag or identification value. Through this, it is possible to distinguish whether there is congestion in only one eye, and to match and determine both the left and right eye regions (220, 225) for the user.

[0068] FIG. 3 is a diagram for explaining an operation of segmenting an eye sub-region image according to one embodiment.

[0069] Referring to FIG. 3, the eye sub-region extraction unit can segment and extract an eye sub-region image using eye sub-region feature points. The eye sub-region image (300) can include feature point (301 to 307) information. The feature points (301 to 307) are acquired by the feature point acquisition unit. However, the eye sub-region cannot be extracted using only the feature point (301 to 307) information. This is because the shape of the eye sub-region appears in the form of a curve and the feature points (301 to 307) are acquired only for some points.

[0070] Therefore, the eye detail region extraction unit can extract the eye detail region image using the coordinate values ​​of the feature points (301 to 307) and a preset interpolation algorithm.

[0071] For example, a preset interpolation algorithm can be set to predict the outer coordinate values ​​of the eye subregion based on the two-dimensional coordinate values ​​of the eye region feature points (301 to 307). For example, the eye region feature points (301 to 307) are expressed as two-dimensional points with coordinate values ​​around the eye region, but an interpolation algorithm can be used to segment the eye subregion in order to extract the eye subregion in the shape of a curve.

[0072] For example, the interpolation algorithm may be a Nearest Neighbor interpolation algorithm, which uses the pixel values ​​of the nearest feature points to determine the outer coordinates of the eye's detailed region. This algorithm is computationally fast, but may result in lower resolution of the boundary. Alternatively, the interpolation algorithm may be a Bilinear interpolation algorithm, which uses the pixel values ​​and distance ratios of four adjacent feature points.

[0073] As another example, the interpolation algorithm could be a bicubic interpolation algorithm, which uses the product of the pixel values ​​of 16 adjacent feature points and a weighted value based on their distance from each other. This algorithm can be used when a sufficient number of feature points are available. As another example, the interpolation algorithm could be a B-Spline interpolation algorithm, which uses the product of the pixel values ​​of 16 adjacent feature points and a weighted value based on their distance from each other to determine the interpolation point, but which is intended to represent a smooth curved shape.

[0074] In addition, interpolation algorithms are not limited to those that find new data values ​​based on given data values. For example, various algorithms can be used, including Forward / Backward Interpolation, Linear Interpolation, Bilinear Interpolation, and Polynomial Interpolation.

[0075] Additionally, interpolation algorithms can be selectively used depending on the number of eye region feature points. For example, the eye region extraction unit can select an interpolation algorithm based on the minimum number of feature points required for each interpolation algorithm, depending on the number of eye region feature points. If two or more interpolation algorithms have the same minimum number of feature points, a specific interpolation algorithm can be selected based on a preset priority among the interpolation algorithms.

[0076] Through this, eye detail region images can be quickly extracted from various image environments.

[0077] The eye sub-region extraction unit can predict and extract coordinate values ​​for an outline connecting feature points using an interpolation algorithm. Through this, the area within the outline can be extracted as an eye sub-region using a line connecting each outline coordinate point. Accordingly, the eye region image (300) can be segmented into an eye sub-region image that includes only the conjunctiva and iris.

[0078] Meanwhile, the conjunctival region image generation unit must detect the iris region in order to generate an image only for the conjunctival region.

[0079] For example, the conjunctival region image generation unit may calculate the sum of gray level values ​​for pixels around a circle set to arbitrary center coordinates and radius values ​​on an eye region image or an eye sub-region image, and may set the circle around which the change in the sum of gray level values ​​is maximum according to the change in the radius value as the boundary of the iris region.

[0080] The iris region is a circular region within the eye. Furthermore, the iris region exhibits significant color variation when compared to the conjunctiva region. Therefore, the conjunctiva region image generation unit can utilize these characteristics to detect the iris region.

[0081] For example, the conjunctival region image generation unit arbitrarily sets a center coordinate in a center coordinate allocation area within a preset range in the eye region image. Once the center coordinate is set, the conjunctival region image generation unit sets an arbitrary radius value within the preset range for the center coordinate and calculates the sum of the gray level values ​​for the pixels around the corresponding circle. The conjunctival region image generation unit calculates the sum of the gray level values ​​for the pixels of the two circles generated while changing the radius value by a preset offset. The conjunctival region image generation unit can store the center coordinate and radius value when the change in the gray level value is greater than a preset standard, and determine the inside of the circle as the iris region. Alternatively, the conjunctival region image generation unit can extract one iris region using only one center coordinate and radius value that have a change in the maximum sum of gray level values.

[0082] FIG. 4 is a drawing for explaining an operation of detecting an iris region according to one embodiment.

[0083] Referring to Fig. 4, the conjunctival region image generation unit can generate iris region candidates (400, 410, 420) having the aforementioned center coordinates and radius values. If three candidates are generated, the conjunctival region image generation unit can determine the interior of the 400th circle with the largest rate of change as the iris region. In Fig. 4, the iris region is generated based on the eye region image, so a case in which a large number of iris region candidates are generated due to the eyelids, etc. is shown. In contrast, when analyzing based on the eye detailed region, the number of iris region candidates may be limited.

[0084] However, if only the maximum rate of change is used as a standard, there is a possibility that the iris region may not be precisely distinguished. Therefore, other embodiments below may also be applied to the present disclosure.

[0085] For example, an iris region candidate can be selected based on a change in the sum of gray level values ​​measured while changing the aforementioned center coordinates and radius values ​​that exceeds a certain threshold. Subsequently, to accurately detect an iris region candidate, the iris region can be corrected based on the center coordinates and finally extracted.

[0086] For example, the iris region can be identified using pupil center information detected using a deep learning model. For example, new boundary points can be extracted based on inaccurate boundaries of the detected iris region and corrected to an elliptical shape to extract a more accurate iris region.

[0087] Referring back to Fig. 4, when three iris region candidates, 400, 410, and 420, are extracted, the conjunctival region image generation unit calculates pupil center information using the DeepEye algorithm. The DeepEye algorithm estimates a mask image indicating the probability that each pixel around the pupil is the pupil region, and calculates the pupil center coordinates through a post-processing operation. Once the pupil center coordinates are calculated, the iris region candidate (400) closest to the corresponding pupil center coordinates among the three iris region candidates (400, 410, and 420) is detected as the iris region.

[0088] The DeepEye model consists of two convolution layers: a Convolution layer and a Strided Convolution layer, along with a Residual layer. The Strided Convolution layer reduces the size of the image. The ASPP (Atrous Spatial Pyramid Pooling) layer simultaneously performs an Atrous Convolution (Dilated Convolution), a 1×1 Convolution, and Max-Pooling on the input image, and then combines all the results. The number of channels is then reduced through the 1×1 Convolution, and finally, the size of the output is made the same as the input through Bilinear Upsampling.

[0089] FIG. 5 is a drawing for explaining an operation of detecting an iris area according to another embodiment.

[0090] Referring to Fig. 5, the conjunctival region image generation unit can detect the pupil region based on the outer line of the pupil using a convex hull algorithm. For example, the conjunctival region image generation unit can detect the iris region using a convex hull algorithm that connects the outermost parts of the given data values ​​on a two-dimensional plane to form a polygonal shape (500).

[0091] The convex hull algorithm is designed to generate a convex polygon containing all points on a two-dimensional coordinate plane using a subset of the points. Thus, the convex hull algorithm is an algorithm that derives the points that form the convex hull when given the coordinates of the points on a two-dimensional coordinate plane. Various algorithms exist for this purpose, and the Graham scan algorithm can be used to derive the outermost line.

[0092] The convex hull algorithm extracts coordinates for pixels with a pixel value greater than or equal to a preset gray level from image information. These coordinates are then arranged on a two-dimensional plane. Next, a reference point is first established among the set coordinates. Typically, the reference point is the one with the smallest y-coordinate. Using this reference point, other points are aligned counterclockwise. Next, lines are drawn between each coordinate in a counterclockwise direction, checking whether CCW is greater than or equal to 0. If it is less than 0, the connection between the corresponding coordinate and the previous coordinate is disconnected, and a different coordinate is selected from the previous coordinates and the same process is repeated. In other words, only connecting lines with CCW greater than or equal to 0 are established, and this determines the outermost line.

[0093] As described above, the conjunctival region image generation unit can detect the iris region in various ways, such as an embodiment that uses the center coordinates and radius values, an embodiment that uses the center coordinates and radius values ​​but finds the center point of the pupil to detect the final iris region, or an embodiment that applies the convex hull algorithm.

[0094] FIG. 6 is a diagram for explaining an operation of generating a conjunctival area image according to one embodiment.

[0095] Referring to FIG. 6, when the iris region is detected, the conjunctival region image generation unit can generate an image of only the conjunctival region (620) by masking the iris region (610) in the eye detailed region image (600).

[0096] For example, the conjunctival region image generation unit can generate a conjunctival region image (620) by performing a bit operation on the iris region (610) and the eye detailed region image (600). The conjunctival region image generation unit can provide data so that only the conjunctival region, which is the white part of the eye, remains by making the overlapping region of the iris region (610) and the eye detailed region image (600) zero. Alternatively, the conjunctival region image generation unit can generate the conjunctival region image (620) by mapping the iris region (610) in the eye detailed region image (600) and zero-masking the iris region (610).

[0097] Through this, all parts except the conjunctival area image (620) are masked in black, enabling accurate judgment of the conjunctival area.

[0098] The conjunctival region image generated through this process can be used as input to a congestion detection algorithm to derive results for determining congestion. By accurately masking only the conjunctival region image and processing it in black, the accuracy of determining congestion in the white portion of the conjunctival region can be increased.

[0099] Additionally, the operation of extracting conjunctival region image information from the aforementioned image information can be applied equally to the image information preprocessing operation used in the learning of the congestion judgment algorithm. That is, the preprocessing operation is performed to extract conjunctival region images from the image information to perform a more accurate learning process. The conjunctival region images that have undergone the preprocessing operation are used as the entire data set of the congestion judgment algorithm.

[0100] For example, a congestion judgment algorithm could be a CNN-based AI algorithm. Examples include AlexNet, ResNet, GoogLeNet, and VGG.

[0101] FIG. 7 is a diagram for explaining a data set of a charge judgment algorithm according to one embodiment.

[0102] Referring to Figure 7, during the learning process of the congestion judgment algorithm, the entire data set can be divided into a learning data set and a validation data set. Additionally, a test data set can be set for some of the data sets. The entire data set can include images of congested conjunctiva areas and images of normal conjunctiva areas.

[0103] The training data set is used to train the congestion detection model. The learner uses the training data set to train different congestion detection models for different epochs. Each different congestion detection model involves slight changes to the hidden layer or hyperparameters.

[0104] A validation dataset is a dataset used to validate a model that has already completed training. A test dataset is a dataset used to evaluate the performance of a model that has completed training and validation.

[0105] For example, the entire data set can be divided into a training data set and a test data set in a ratio of 8:2, or a training data set, a test data set, and a validation data set in a ratio of 6:2:2.

[0106] What the validation and test data sets have in common is that they do not train a model. However, while the validation data set does not train the model, it does contribute to its learning.

[0107] For example, the first congestion detection model trains by changing the epoch using the training data set. For each epoch, validation is performed using the validation data set. In the case of AI models, training with too many epochs can lead to overfitting and increased error. To prevent this, accuracy is assessed using the validation data set for each epoch. When the minimum error rate (highest accuracy) is achieved on the validation data set, further training is halted by stopping the epoch increase for that model. The congestion detection model trained in this way can be applied to a congestion detection algorithm.

[0108] Similarly, for the second ... Nth congestion judgment model with some modifications to the hyperparameters and hidden layers, the accuracy is measured with the validation data set while increasing the epoch.

[0109] Through the aforementioned actions, the congestion judgment model is determined using the epoch, hyperparameters, and hidden layer of the model with the highest accuracy as measured through the verification data set.

[0110] The ocular congestion detection device described above can improve the accuracy of determining ocular congestion by preprocessing images captured with a popular camera device, such as an RGB camera. Furthermore, by extracting and masking only the necessary portions of the image during the image preprocessing process, it can achieve rapid processing speed.

[0111] Below, the ocular congestion detection method, in which the operations of each of the aforementioned embodiments are performed, is briefly described again with reference to the drawings. The method described below can perform some or all of the aforementioned operations.

[0112] Fig. 8 is a flowchart for explaining a method for detecting ocular congestion according to one embodiment.

[0113] Referring to FIG. 8, the ocular congestion detection method for detecting whether the ocular congestion is present may include an image acquisition step for acquiring image information (S810).

[0114] For example, the image acquisition step may receive image information captured by a sensor such as a camera. Here, the camera refers to a general camera such as the aforementioned RGB camera. Alternatively, the image acquisition step may acquire image information by capturing frames from an image.

[0115] The method for detecting ocular congestion may include a feature point acquisition step for acquiring ocular region feature points of an ocular region image extracted from image information (S820).

[0116] The feature point acquisition step can acquire an eye region image including the eye from the acquired image information. For example, the eye region image can be configured to include eye region feature points but exclude other region feature points included in the facial image information. For example, the image information can include a background, a person's facial image information, and other body images such as a neck. Once the image information is acquired, the feature point acquisition step can extract the person's facial image information from the image information using an artificial intelligence algorithm. For this purpose, a landmark extractor that acquires the facial image information from the image information can be used. The landmark extractor performs the function of extracting feature points related to the human facial contour from the image.

[0117] Once facial image information is extracted, facial image information including the eye region can be identified. As described above, the facial image information can be configured to include only feature points assigned to the eye region. Through this, the feature point acquisition step can acquire eye region feature points from the image information.

[0118] The method for detecting ocular congestion may include an ocular detailed region extraction step of segmenting and extracting an ocular detailed region image using ocular region feature points (S830).

[0119] An eye region image contains features specific to the eye region, but may also include images of the surrounding skin and eyebrows. Therefore, to accurately determine whether or not bloodshot eyes are present, the eye region extraction step must extract only the eye region images.

[0120] To achieve this, the eye region extraction step can utilize eye region feature points. The eye region extraction step can use the acquired eye region feature points and an image processing algorithm to determine the outline of the eye region. Once the outline is determined, the eye region extraction step can extract only the portion within the outline as an eye region image.

[0121] For example, a preset interpolation algorithm can be set to extract an eye sub-region image by predicting the outer coordinate values ​​of the eye sub-region based on the two-dimensional coordinate values ​​of the eye region feature points. For example, if the two-dimensional coordinate values ​​for N eye region feature points are confirmed, the interpolation algorithm can predict the coordinate values ​​between the eye region feature points using the two-dimensional coordinate values. For example, various algorithms such as Forward / Backward Interpolation, Linear Interpolation, Bilinear Interpolation, and Polynomial Interpolation can be used as the interpolation algorithm. Through this, the eye sub-region extraction step can obtain the outer coordinate values ​​connecting the eye region feature points and extract the outline of the eye region by connecting them.

[0122] In the eye detail region extraction step, once the outline is extracted, an image can be extracted based on this to create an image only for the eye detail region within the outline.

[0123] The ocular congestion detection method may include a conjunctival region image generation step of detecting an iris region from an ocular region image or an ocular detailed region image and generating a conjunctival region image using the ocular detailed region image and the iris region (S840).

[0124] The conjunctival region image generation step can generate a conjunctival region image by isolating only the conjunctival region, which serves as a criterion for determining congestion. To achieve this, the conjunctival region image generation step must detect the iris region from the eye's detailed regions.

[0125] For example, the conjunctival region image generation step may calculate the sum of gray level values ​​for pixels around a circle set to arbitrary center coordinates and radius values ​​in the eye region image or the eye sub-region image, and set the circumference of the circle where the change in the sum of gray level values ​​is maximum according to the change in the radius value as the boundary of the iris region. For example, the conjunctival region image generation step may set arbitrary coordinates in the eye region image or the eye sub-region image as the center coordinates, set the radius value to an arbitrary value, and calculate the sum of gray level values ​​for pixels around the circle for each arbitrary center coordinate and each arbitrary radius value.

[0126] The conjunctival region image generation step can continuously calculate the sum of these gray level values ​​while changing the center coordinate and radius values. The conjunctival region image generation step sets the circumference of the circle where the change in the sum of the calculated gray level values ​​is the maximum as the boundary of the iris region, and stores the center coordinate and radius values ​​at this time. For example, the conjunctival region image generation step can determine the center coordinate using the experimental range of the center coordinate of the eye during the generation of the eye region image or the eye sub-region image. That is, the center point of the iris (i.e., the center point of the eye) in the eye region image or the eye sub-region image is formed in the central part of the image. Therefore, for fast computational processing, the conjunctival region image generation step first arbitrarily sets the center coordinate within a preset coordinate range. Similarly, the radius value can also be set to a range of arbitrary values ​​so that it is first selected within a certain range depending on the image scale, etc. Through this operation, the conjunctival region image generation step can detect the iris region more quickly.

[0127] As another example, the conjunctival region image generation step can determine a certain number of candidate iris regions using a circle detection algorithm, determine the pupil center point using the DeepEye model, and select the iris region corresponding to the pupil center point from the candidate iris regions to determine the final iris region. Through this, a more accurate iris region can be detected in cases where multiple candidate iris regions appear through the circle detection algorithm, such as eyelashes. The circle detection algorithm can be an algorithm that detects by utilizing changes in pixels while changing the aforementioned center coordinates and radius values.

[0128] As another example, the conjunctival region image generation step can detect the iris region by applying a convex hull algorithm to the eye region image or the eye sub-region image. For example, the conjunctival region image generation step can detect the iris region using the convex hull algorithm, which connects the outermost parts of a plurality of data values ​​and represents them in the form of a polygon using the eye sub-region image. The shape of the iris may not appear circular due to the eyelids of the human body, etc. Therefore, the iris region detection using the center coordinates and radius values ​​described above may incur errors depending on the situation. In particular, when the iris region is detected using the eye sub-region image instead of the eye region image, the iris is likely to not appear circular. Therefore, the conjunctival region image generation step can more accurately obtain the shape of the iris region by using an image processing algorithm such as the convex hull. For example, the conjunctival region image generation step can represent the outermost part of the given data values ​​(so that all the given data values ​​are included) on the two-dimensional plane of the eye detailed region image as a polygon, and detect the interior of the polygon as the iris region.

[0129] In addition to the aforementioned operations, the conjunctival region image generation step can detect the iris region using an image processing algorithm or artificial intelligence algorithm capable of detecting the iris region.

[0130] Once the iris region is detected, the conjunctival region image generation step can generate a conjunctival region image by excluding the iris region from the eye detailed region image. In other words, the conjunctival region image generation step can generate a conjunctival region image by masking the iris region from the eye detailed region image that includes only the conjunctiva and iris regions, leaving only the conjunctival region.

[0131] For example, the conjunctival region image generation step can generate a conjunctival region image by performing bit operations on the iris region and eye sub-region images. For example, the conjunctival region image generation step can generate a conjunctival region image so that only the conjunctival region appears by performing bit operations on the detected iris region and eye sub-region images. In other words, the conjunctival region image generation step can generate a conjunctival region image so that only the conjunctival region remains by making the overlapping area between the eye sub-region image and the iris region 0 through image bit operations.

[0132] As another example, the conjunctival region image generation step can generate a conjunctival region image by zero-masking the iris region in the eye detailed region image so that only the conjunctival region remains.

[0133] The method for detecting ocular congestion may include a judgment step of inputting an image of the conjunctival region into a pre-learned congestion judgment algorithm to derive a classification result value for whether or not there is congestion (S850).

[0134] For example, the pre-learned congestion judgment algorithm is an artificial intelligence algorithm learned using the entire data set including congested conjunctiva area image data and normal conjunctiva area image data.

[0135] For example, the pre-trained congestion detection algorithm is an algorithm trained using a CNN-based artificial intelligence algorithm. To improve the accuracy of congestion detection, a conjunctival region image can be extracted from the aforementioned image information to build a dataset for the congestion detection algorithm. For example, once the image information is acquired, the congestion detection algorithm learner performs a preprocessing step in which it extracts an eye sub-region image using feature points from the eye region image, and then extracts a conjunctival region image using the iris region and eye sub-region images.

[0136] Afterwards, when the conjunctival region image is extracted, the learning machine learns the congestion judgment algorithm using the entire data set including conjunctival region image data and normal conjunctival region image data.

[0137] Additionally, the entire dataset is divided into a training dataset used during the learning process and a validation dataset used to calculate accuracy while changing at least one of the epoch, hyperparameter, and hidden layer. The validation dataset is used during the learning process but not in the actual learning itself. It is used to verify the learning using the training dataset. Because it is used during the learning process, it is composed of a different dataset from a typical test set.

[0138] While the learner is learning using the training data set, the optimal epoch that avoids overfitting can be identified by increasing the number of epochs and calculating the accuracy of the validation data set. Similarly, while learning using the training data set, the optimal number of parameters and layers can be determined by varying the number of hyperparameters or hidden layers and calculating the accuracy of the validation data set.

[0139] Additionally, in order to verify the validity of the data set during the learning process, an operation may be performed to verify the validity of using the preprocessed conjunctival region image by applying the image information in RGB format before preprocessing to an algorithm such as Grad-CAM, which is an explainable artificial intelligence.

[0140] The judgment step inputs the conjunctival region image as input to the pre-learned congestion judgment algorithm learned through the aforementioned process, and obtains a classification value for congestion as output to determine whether the eye is congested or not in the image.

[0141] Each of the aforementioned steps may perform some or all of the operations described with reference to FIGS. 1 through 7, and the execution steps may involve two or more steps being performed simultaneously or in a slightly different order. Furthermore, specific steps may be combined into a single step, or specific steps may be separated into two or more steps.

[0142] The above description is merely an example of the technical idea of ​​the present disclosure, and those skilled in the art to which the present disclosure pertains will appreciate that various modifications and variations can be made without departing from the essential characteristics of the technical idea of ​​the present disclosure. In addition, the present embodiments are not intended to limit the technical idea of ​​the present disclosure but rather to explain it, and therefore the scope of the technical idea of ​​the present disclosure is not limited by these embodiments. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present disclosure.

Claims

1. Image acquisition unit for acquiring image information; A feature point acquisition unit for acquiring eye region feature points of an eye region image extracted from the above image information; An eye region extraction unit that segments and extracts an eye region image using the above eye region feature points; A conjunctival region image generation unit detecting an iris region from the eye region image or the eye detail region image and generating a conjunctival region image using the eye detail region image and the iris region; and An ocular congestion detection device including a judgment unit that inputs the conjunctival region image into a pre-learned congestion judgment algorithm and derives a classification result value regarding congestion.

2. In paragraph 1, The above eye area image is, An ocular congestion detection device configured to include the above ocular region feature points, but exclude other region feature points included in the facial image information.

3. In paragraph 1, The above eye detail region extraction part is, An ocular congestion detection device that extracts an ocular detailed region image using the ocular region feature points and a preset interpolation algorithm.

4. In paragraph 3, The above preset interpolation algorithm is, An ocular congestion detection device configured to extract an ocular sub-region image by predicting the outer coordinate values ​​of the ocular sub-region based on the two-dimensional coordinate values ​​of the ocular region feature points.

5. In paragraph 1, The above conjunctival area image generation unit, An ocular congestion detection device that calculates the sum of gray level values ​​for pixels around a circle set to arbitrary center coordinates and radius values ​​on the eye region image or the eye sub-region image, and sets the circle around which the change in the sum of the gray level values ​​is maximum according to the change in the radius value as the boundary of the iris region.

6. In paragraph 1, The above conjunctival area image generation unit, An ocular congestion detection device that detects the iris region by applying a convex hull algorithm to the ocular region image or the ocular sub-region image.

7. In paragraph 1, The above conjunctival area image generation unit, An ocular congestion detection device that generates an image of the conjunctiva region only from the image of the ocular detailed region, excluding the iris region.

8. In paragraph 7, The above conjunctival area image generation unit, An eye congestion detection device that generates an image of the conjunctiva region by performing bit operations on the images of the iris region and the eye subregion.

9. In paragraph 1, The above pre-learned congestion judgment algorithm is It is an artificial intelligence algorithm trained using the entire data set including image data of conjunctival conjunctiva area and image data of normal conjunctiva area. An ocular congestion detection device in which the entire data set is divided into a learning data set used in the learning process and a validation data set for calculating accuracy while changing at least one of the epoch, hyperparameter, and hidden layer in the learning process.

10. In a method for detecting eye congestion, which detects whether the eye is congested, Image acquisition step for acquiring image information; A feature point acquisition step for acquiring eye region feature points of an eye region image extracted from the above image information; An eye sub-region extraction step for segmenting and extracting an eye sub-region image using the above eye region feature points; A conjunctival region image generation step for detecting an iris region from the eye region image or the eye detail region image and generating a conjunctival region image using the eye detail region image and the iris region; and A method for detecting ocular congestion, comprising a judgment step of inputting the conjunctival region image into a pre-learned congestion judgment algorithm to derive a classification result value for whether or not there is congestion.

11. In Article 10, The above eye area image is, A method for detecting ocular congestion, wherein the above ocular region feature points are included, but other region feature points included in the facial image information are excluded.

12. In paragraph 10, The above eye detail region extraction step is, A method for detecting ocular congestion by extracting an ocular detailed region image using the above ocular region feature points and a preset interpolation algorithm.

13. In paragraph 12, The above preset interpolation algorithm is, A method for detecting ocular congestion, wherein the method is set to extract an image of an ocular subregion by predicting the outer coordinate values ​​of the ocular subregion based on the two-dimensional coordinate values ​​of the feature points of the ocular region.

14. In paragraph 10, The above conjunctival area image generation step is, A method for detecting ocular congestion, comprising: calculating the sum of gray level values ​​for pixels around a circle set to arbitrary center coordinates and radius values ​​on the eye region image or the eye sub-region image; and setting the circle around which the change in the sum of gray level values ​​is maximum according to the change in the radius value as the boundary of the iris region.

15. In paragraph 10, The above conjunctival area image generation step is, A method for detecting ocular congestion by applying a convex hull algorithm to the ocular region image or the ocular sub-region image to detect the iris region.

16. In paragraph 10, The above conjunctival area image generation step is, A method for detecting eye congestion by bit-operating the iris region and the eye sub-region images to generate the conjunctiva region image.

17. In paragraph 10, The above pre-learned congestion judgment algorithm is It is an artificial intelligence algorithm trained using the entire data set including image data of conjunctival conjunctiva area and image data of normal conjunctiva area. A method for detecting ocular congestion, wherein the above entire data set is divided into a learning data set used in the learning process and a validation data set for calculating accuracy while changing at least one of the epoch, hyperparameter, and hidden layer in the learning process.

Images