Method for estimating exophthalmometric value, and system for performing same

Abstract

According to one embodiment disclosed according to the present application, a method for estimating an exophthalmometric value can be provided, the method comprising: acquiring an image in which at least one eye of a subject appears, the image including a plurality of pixels to which a value corresponding to brightness and / or color is allocated; performing prestored preprocessing on the image so as to acquire a preprocessed image; using the preprocessed image in a pretrained depth image generation model so as to acquire a depth image corresponding to the preprocessed image, the depth image including a plurality of pixels so that a depth value indicating the relative distance of an object corresponding to each of the pixels of the preprocessed image is allocated to each of the plurality of pixels of the depth image; and applying both the preprocessed image and the depth image to a pretrained exophthalmometric value estimation model so as to estimate an exophthalmometric value for the eye of the subject.

Description

Method for estimating ocular protrusion values ​​and system for performing the same

[0001] The present invention relates to a method for estimating a protrusion value of an eye of a subject using an image of the subject's eye captured by a camera, and a system for performing the same.

[0002] Exophthalmos is a condition in which the eyeball protrudes forward more than normal. It can be caused by various factors, such as thyroid dysfunction or tumors. The degree of exophthalmos varies from person to person, and the need and direction of treatment vary depending on the exophthalmos value, so accurate measurement is important. Furthermore, after treatment of exophthalmos, continuous monitoring and management of the exophthalmos value is necessary to prevent recurrence. Therefore, knowing the patient's exophthalmos value is necessary for appropriate treatment and management, but the problem with knowing the patient's exophthalmos value is that it can only be measured by a medical professional at a hospital.

[0003] To address these issues, attempts have been made to estimate ocular protrusion values ​​using images taken by users of their own eyes. For example, a user takes a photo of the side of their eye using their mobile device, and attempts have been made to estimate ocular protrusion values ​​using the side image. While side images are useful for estimating ocular protrusion values, estimating ocular protrusion values ​​from side images requires capturing side images that satisfy predefined conditions. However, capturing a side image that satisfies these conditions requires the user to look straight ahead and capture the side of their eye from a direction perpendicular to the user. This makes capturing a side image that satisfies these conditions quite difficult and causes inconvenience to users.

[0004] Accordingly, there have been attempts to estimate ocular protrusion values ​​using frontal images, rather than side images captured by users on mobile devices. Frontal images offer the advantage of being easily captured by users themselves using mobile devices. However, unlike side images, frontal images are two-dimensional images in which the degree of eye protrusion cannot be visually confirmed. Therefore, a special method was needed to estimate ocular protrusion values ​​using frontal images.

[0005] Based on the assumption that contour information obtained from the frontal image can be utilized to estimate the ocular protrusion value, the ocular protrusion value was calculated by calculating the z-axis difference from the center of the eye to the outer corner of the eye using a 3D facial landmark detection model that recognizes the 3D coordinate values ​​of facial landmarks. However, using the 3D facial landmark detection model alone did not yield an accurate ocular protrusion value.

[0006] Meanwhile, based on the assumption that depth information obtained from the frontal image could be utilized to estimate the ocular protrusion value, a depth estimation model that estimates the depth value of the face was used to generate a depth map of the eye image, thereby calculating the ocular protrusion value. However, using the depth map alone did not yield accurate ocular protrusion values.

[0007] Accordingly, a method was needed to estimate the ocular protrusion value using a facial image of the user's eyes captured by the user's mobile device.

[0008]

[0009] The problem disclosed by the present application is to provide a method for estimating ocular protrusion values ​​using a frontal facial image acquired using a personal electronic device that can be used by the general public, rather than a specialized medical diagnostic device. Specifically, the present invention provides a method for generating a depth image using the acquired frontal facial image, and estimating ocular protrusion values ​​using an estimation model utilizing both the face image and the depth image.

[0010] Another problem that the present invention seeks to solve is to provide medical information to a user based on the estimated eye protrusion value described above.

[0011] The problems to be solved by this application are not limited to the problems described above, and problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the technology disclosed by this application belongs from this specification and the attached drawings.

[0012]

[0013] According to one embodiment disclosed by the present application, a method is provided, comprising: acquiring a first visible-light image captured by a visible-light camera, wherein at least one eye of a subject appears in the first visible-light image, wherein the first visible-light image includes a plurality of pixels, and wherein each of the plurality of pixels of the first visible-light image is assigned a value corresponding to at least one of brightness and color; generating a first depth image corresponding to the first visible-light image using a first artificial neural network model that has been trained in advance, wherein the first artificial neural network model is trained to output a generated image in which an estimated depth value corresponding to at least one pixel included in an input image is reflected in at least one pixel, and the first depth image includes a plurality of pixels, and wherein each of the plurality of pixels of the first depth image is assigned a depth value estimated based on the first visible-light image; performing preprocessing on the first visible-light image to generate a preprocessed first visible-light image; performing preprocessing on the first depth image to generate a preprocessed first depth image; And a method for predicting ocular protrusion may be provided, including applying both the preprocessed first visible light image and the preprocessed first depth image to a pre-learned ocular protrusion value estimation model to estimate an ocular protrusion value for the eye of the subject, wherein the ocular protrusion value estimation model is learned using a preprocessed second visible light image generated by preprocessing a second visible light image, a preprocessed second depth image generated by preprocessing a second depth image corresponding to the second visible light image, and an ocular protrusion value corresponding to the second visible light image.

[0014] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, including: obtaining an image in which at least one eye of a subject appears, the image including a plurality of pixels to which a value corresponding to at least one of brightness and color is assigned; performing pre-stored preprocessing on the image to obtain a pre-processed image; obtaining a depth image corresponding to the pre-processed image by using the pre-processed image in a pre-learned depth image generation model, the depth image including a plurality of pixels, each of the plurality of pixels of the depth image being assigned a depth value representing a relative distance of an object corresponding to each of the pixels of the pre-processed image; and applying both the pre-processed image and the depth image to a pre-learned ocular protrusion value estimation model to estimate an ocular protrusion value for the eye of the subject.

[0015] According to one embodiment disclosed by the present application, a method for estimating an eye protrusion value may be provided, wherein the eye protrusion value estimation model is learned by using a learning preprocessed image generated by preprocessing an image in which at least one eye of a first subject appears, a learning depth image corresponding to the learning preprocessed image, and an eye protrusion value of the first subject.

[0016] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the image includes one eye of the subject, and the ocular protrusion value is a protrusion value of the eye appearing in the image.

[0017] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the image includes both eyes of the subject, and the ocular protrusion value is a set of a protrusion value of the left eye of the subject and a protrusion value of the right eye of the subject appearing in the image.

[0018] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the image includes both eyes of the subject, the preprocessed image includes a first image corresponding to a left eye and a second image corresponding to a right eye among both eyes of the subject, and an ocular protrusion value estimated by applying the first image is a protrusion value of the right eye of the subject appearing in the image, and an ocular protrusion value estimated by applying the second image is a protrusion value of the left eye of the subject appearing in the image.

[0019] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the image shows an entire face area including both eyes of the subject, and performing preprocessing on the image to generate the preprocessed image includes cropping the image to at least a portion of the area where both eyes of the subject appear to generate the preprocessed image.

[0020] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the image shows an entire face area including at least one eye and a bridge of the nose of the subject, and performing preprocessing on the image to generate the preprocessed image includes cropping the image to at least a portion of an area where the one eye and the bridge of the nose of the subject appear to generate the preprocessed image.

[0021] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the depth value is a value between a preset minimum value and a preset maximum value, and is a maximum value if an object represented in at least one pixel of the preprocessed image corresponding to a pixel of the depth image to which the depth value is assigned is a closest object appearing in the preprocessed image, is a minimum value if the object is a farthest object appearing in the preprocessed image, and is a value closer to the maximum value when the object is a relatively closer object in the preprocessed image, and is a value closer to the minimum value when the object is a relatively farther object in the preprocessed image.

[0022] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the depth value is a value between a preset minimum value and a preset maximum value, and the depth value is a minimum value if an object represented in at least one pixel of the preprocessed image corresponding to a pixel of the depth image to which the depth value is assigned is a closest object appearing in the preprocessed image, and the depth value is a maximum value if the object is a farthest object appearing in the preprocessed image, and the closer the object is to the preprocessed image, the closer the value is to the minimum value, and the closer the object is to the preprocessed image, the closer the value is to the maximum value.

[0023] According to one embodiment disclosed by the present application, a method for estimating an eye protrusion value may be provided, wherein the eye protrusion value estimation model has an artificial neural network structure, processes values ​​assigned to each pixel of the preprocessed image to obtain a first intermediate result, processes values ​​assigned to each pixel of the depth image to obtain a second intermediate result, connects the first intermediate result and the second intermediate result to obtain a third intermediate result, and processes the third intermediate result to output the eye protrusion value.

[0024] According to one embodiment disclosed by the present application, a method for estimating an eye protrusion value may be provided, wherein the eye protrusion value estimation model processes values ​​assigned to each pixel of the preprocessed image with a first layer having a first ResNet structure to obtain a first intermediate result, processes values ​​assigned to each pixel of the depth image with a second layer having the first ResNet structure to obtain a second intermediate result, connects the first intermediate result and the second intermediate result to obtain a third intermediate result, and processes the third intermediate result with a third layer having a second ResNet structure to output the eye protrusion value.

[0025] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the image shows an entire face area of ​​the subject including both eyes of the subject, and the image satisfies at least one of the following conditions: i) a degree of smile of the face is within a predetermined level, ii) a left-right rotation angle of the face is within a predetermined angular range, iii) a vertical rotation angle of the face is within a predetermined angular range, and iv) the face is located within a predetermined distance.

[0026] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein obtaining the image captured by the visible light camera further includes providing a shooting guide for obtaining the visible light image satisfying the condition.

[0027] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, further comprising: providing the estimated ocular protrusion value to a user device.

[0028] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, further comprising: providing a visit guidance message for the subject if the estimated ocular protrusion value is greater than or equal to a preset threshold value.

[0029] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the threshold value is determined based on at least one of the race and facial shape of the subject.

[0030] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, further comprising: obtaining a past ocular protrusion value for the subject; and providing a visit guidance message for the subject if a difference between the estimated ocular protrusion value and the past ocular protrusion value is greater than or equal to a preset threshold value.

[0031] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, further comprising: providing information to determine at least one of thyroid ophthalmopathy severity and thyroid ophthalmopathy activity for the subject based on the estimated ocular protrusion value.

[0032] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, further comprising: providing information to determine whether at least one of drug treatment and surgery is required for the subject based on the estimated ocular protrusion value.

[0033] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, further comprising: providing information to determine whether at least one of drug treatment and surgery is required for the subject based on the estimated ocular protrusion value.

[0034] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein acquiring the image comprises acquiring at least a first image and a second image for the object, wherein the first image and the second image are acquired under the same shooting conditions.

[0035] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein acquiring the preprocessed image, acquiring the depth image, and estimating the ocular protrusion value are performed once for the first image and once for the second image, and an average value of the first ocular protrusion value for the first image and the second ocular protrusion value for the second image is estimated as the ocular protrusion value on the day the image was acquired. The problem-solving means of the present invention is not limited to the above-described solutions, and solutions that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the present specification and the accompanying drawings.

[0036]

[0037] According to one embodiment disclosed by the present application, a method for estimating ocular protrusion values ​​using a frontal facial image acquired using a personal electronic device that can be used by the general public rather than a professional medical diagnostic device may be provided. Specifically, a method may be provided for generating a depth image using the acquired frontal facial image, and estimating ocular protrusion values ​​using an estimation model utilizing both the face image and the depth image.

[0038] According to one embodiment disclosed by the present application, medical information can be provided to a user based on the estimated eye protrusion value described above.

[0039] The effects of the present invention are not limited to the effects described above, and effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from this specification and the attached drawings.

[0040]

[0041] FIG. 1 is a diagram for explaining a process for estimating an eye protrusion value according to one embodiment.

[0042] FIG. 2 is a drawing for explaining a visible light image and a depth image according to one embodiment.

[0043] FIG. 3 is a diagram for explaining an eye protrusion value estimation model according to one embodiment.

[0044] FIG. 4 is a diagram for explaining a process for estimating an eye protrusion value according to one embodiment.

[0045] FIG. 5 is a diagram for explaining a process for estimating an eye protrusion value according to one embodiment.

[0046] FIG. 6 is a diagram illustrating a system for estimating an eye protrusion value according to one embodiment.

[0047] Fig. 7 is a flowchart illustrating a process for estimating an eye protrusion value according to one embodiment.

[0048] Figure 8 is a drawing for explaining a shooting guide according to one embodiment.

[0049] FIG. 9 is a drawing for explaining a shooting guide according to one embodiment.

[0050] FIG. 10 is a drawing for explaining a shooting guide according to one embodiment.

[0051] Fig. 11 is a drawing for explaining a preprocessing method of a visible light image according to one embodiment.

[0052] FIG. 12 is a drawing for explaining a preprocessing method of a visible light image according to one embodiment.

[0053] Fig. 13 is a drawing for explaining a preprocessing method of a visible light image according to one embodiment.

[0054] FIG. 14 is a drawing for explaining a preprocessing method of a visible light image according to one embodiment.

[0055] FIG. 15 is a drawing for explaining a UI that provides an eye protrusion value according to one embodiment.

[0056] FIG. 16 is a drawing for explaining a UI that provides a graph of eye protrusion values ​​according to one embodiment.

[0057] Fig. 17 is a flowchart illustrating a process for estimating an eye protrusion value according to one embodiment.

[0058] Fig. 18 is a flowchart illustrating a process for estimating an eye protrusion value according to one embodiment.

[0059] Since the embodiments disclosed by this application are intended to clearly explain the idea of ​​the present invention to a person having ordinary skill in the art to which the present invention pertains, the present invention is not limited to the embodiments disclosed by this application, and the scope of the present invention should be interpreted to include modified or varied examples that do not depart from the idea of ​​the present invention.

[0060] The terms used in this application have been selected from widely used, common terms, taking into account the functions of the present invention. However, these terms may vary depending on the intentions of those skilled in the art, customs, or the emergence of new technologies. However, if a specific term is defined and used with an arbitrary meaning, the meaning of that term will be described separately. Therefore, the terms used in this specification should be interpreted based on their actual meaning and the overall content of this application, rather than simply their names.

[0061] The numbers used in the description of this application (e.g., first and second, etc.) are merely identifiers to distinguish one component from another.

[0062] In the examples below, singular expressions include plural expressions unless the context clearly indicates otherwise.

[0063] In the following examples, terms such as “include” or “have” mean that a feature or component disclosed in the present application is present, and do not preclude the possibility that one or more other features or components may be added.

[0064] The drawings attached to this application are intended to facilitate explanation of the present disclosure, and shapes depicted in the drawings may be exaggerated as necessary to help understanding of the present disclosure, and the present disclosure is not limited by the drawings.

[0065] In some embodiments, where implementations are otherwise feasible, specific process sequences may be performed in a different order than described. For example, two processes described in succession may be performed substantially simultaneously, or in a reverse order from the described order.

[0066] In the present application, if it is determined that a specific description of the configuration or function of a public notice related to the present invention may obscure the gist of the present invention, a detailed description thereof may be omitted as necessary.

[0067] According to one embodiment disclosed by the present application, a method is provided, comprising: acquiring a first visible-light image captured by a visible-light camera, wherein at least one eye of a subject appears in the first visible-light image, wherein the first visible-light image includes a plurality of pixels, and wherein each of the plurality of pixels of the first visible-light image is assigned a value corresponding to at least one of brightness and color; generating a first depth image corresponding to the first visible-light image using a first artificial neural network model that has been trained in advance, wherein the first artificial neural network model is trained to output a generated image in which an estimated depth value corresponding to at least one pixel included in an input image is reflected in at least one pixel, and the first depth image includes a plurality of pixels, and wherein each of the plurality of pixels of the first depth image is assigned a depth value estimated based on the first visible-light image; performing preprocessing on the first visible-light image to generate a preprocessed first visible-light image; performing preprocessing on the first depth image to generate a preprocessed first depth image; And a method for predicting ocular protrusion may be provided, including applying both the preprocessed first visible light image and the preprocessed first depth image to a pre-learned ocular protrusion value estimation model to estimate an ocular protrusion value for the eye of the subject, wherein the ocular protrusion value estimation model is learned using a preprocessed second visible light image generated by preprocessing a second visible light image, a preprocessed second depth image generated by preprocessing a second depth image corresponding to the second visible light image, and an ocular protrusion value corresponding to the second visible light image.

[0068] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, including: obtaining an image in which at least one eye of a subject appears, the image including a plurality of pixels to which a value corresponding to at least one of brightness and color is assigned; performing pre-stored preprocessing on the image to obtain a pre-processed image; obtaining a depth image corresponding to the pre-processed image by using the pre-processed image in a pre-learned depth image generation model, the depth image including a plurality of pixels, each of the plurality of pixels of the depth image being assigned a depth value representing a relative distance of an object corresponding to each of the pixels of the pre-processed image; and applying both the pre-processed image and the depth image to a pre-learned ocular protrusion value estimation model to estimate an ocular protrusion value for the eye of the subject.

[0069] According to one embodiment disclosed by the present application, a method for estimating an eye protrusion value may be provided, wherein the eye protrusion value estimation model is learned by using a learning preprocessed image generated by preprocessing an image in which at least one eye of a first subject appears, a learning depth image corresponding to the learning preprocessed image, and an eye protrusion value of the first subject.

[0070] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the image includes one eye of the subject, and the ocular protrusion value is a protrusion value of the eye appearing in the image.

[0071] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the image includes both eyes of the subject, and the ocular protrusion value is a set of a protrusion value of the left eye of the subject and a protrusion value of the right eye of the subject appearing in the image.

[0072] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the image includes both eyes of the subject, the preprocessed image includes a first image corresponding to a left eye and a second image corresponding to a right eye among both eyes of the subject, and an ocular protrusion value estimated by applying the first image is a protrusion value of the right eye of the subject appearing in the image, and an ocular protrusion value estimated by applying the second image is a protrusion value of the left eye of the subject appearing in the image.

[0073] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the image shows an entire face area including both eyes of the subject, and performing preprocessing on the image to generate the preprocessed image includes cropping the image to at least a portion of the area where both eyes of the subject appear to generate the preprocessed image.

[0074] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the image shows an entire face area including at least one eye and a bridge of the nose of the subject, and performing preprocessing on the image to generate the preprocessed image includes cropping the image to at least a portion of an area where the one eye and the bridge of the nose of the subject appear to generate the preprocessed image.

[0075] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the depth value is a value between a preset minimum value and a preset maximum value, and is a maximum value if an object represented in at least one pixel of the preprocessed image corresponding to a pixel of the depth image to which the depth value is assigned is a closest object appearing in the preprocessed image, is a minimum value if the object is a farthest object appearing in the preprocessed image, and is a value closer to the maximum value when the object is a relatively closer object in the preprocessed image, and is a value closer to the minimum value when the object is a relatively farther object in the preprocessed image.

[0076] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the depth value is a value between a preset minimum value and a preset maximum value, and the depth value is a minimum value if an object represented in at least one pixel of the preprocessed image corresponding to a pixel of the depth image to which the depth value is assigned is a closest object appearing in the preprocessed image, and the depth value is a maximum value if the object is a farthest object appearing in the preprocessed image, and the closer the object is to the preprocessed image, the closer the value is to the minimum value, and the closer the object is to the preprocessed image, the closer the value is to the maximum value.

[0077] According to one embodiment disclosed by the present application, a method for estimating an eye protrusion value may be provided, wherein the eye protrusion value estimation model has an artificial neural network structure, processes values ​​assigned to each pixel of the preprocessed image to obtain a first intermediate result, processes values ​​assigned to each pixel of the depth image to obtain a second intermediate result, connects the first intermediate result and the second intermediate result to obtain a third intermediate result, and processes the third intermediate result to output the eye protrusion value.

[0078] According to one embodiment disclosed by the present application, a method for estimating an eye protrusion value may be provided, wherein the eye protrusion value estimation model processes values ​​assigned to each pixel of the preprocessed image with a first layer having a first ResNet structure to obtain a first intermediate result, processes values ​​assigned to each pixel of the depth image with a second layer having the first ResNet structure to obtain a second intermediate result, connects the first intermediate result and the second intermediate result to obtain a third intermediate result, and processes the third intermediate result with a third layer having a second ResNet structure to output the eye protrusion value.

[0079] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the image shows an entire face area of ​​the subject including both eyes of the subject, and the image satisfies at least one of the following conditions: i) a degree of smile of the face is within a predetermined level, ii) a left-right rotation angle of the face is within a predetermined angular range, iii) a vertical rotation angle of the face is within a predetermined angular range, and iv) the face is located within a predetermined distance.

[0080] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein obtaining the image captured by the visible light camera further includes providing a shooting guide for obtaining the visible light image satisfying the condition.

[0081] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, further comprising: providing the estimated ocular protrusion value to a user device.

[0082] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, further comprising: providing a visit guidance message for the subject if the estimated ocular protrusion value is greater than or equal to a preset threshold value.

[0083] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein the threshold value is determined based on at least one of the race and facial shape of the subject.

[0084] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, further comprising: obtaining a past ocular protrusion value for the subject; and providing a visit guidance message for the subject if a difference between the estimated ocular protrusion value and the past ocular protrusion value is greater than or equal to a preset threshold value.

[0085] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, further comprising: providing information to determine at least one of thyroid ophthalmopathy severity and thyroid ophthalmopathy activity for the subject based on the estimated ocular protrusion value.

[0086] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, further comprising: providing information to determine whether at least one of drug treatment and surgery is required for the subject based on the estimated ocular protrusion value.

[0087] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, further comprising: providing information to determine whether at least one of drug treatment and surgery is required for the subject based on the estimated ocular protrusion value.

[0088] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein acquiring the image comprises acquiring at least a first image and a second image for the object, wherein the first image and the second image are acquired under the same shooting conditions.

[0089] According to one embodiment disclosed by the present application, a method for estimating an ocular protrusion value may be provided, wherein acquiring the preprocessed image, acquiring the depth image, and estimating the ocular protrusion value are performed once for the first image and once for the second image, and an average value of the first ocular protrusion value for the first image and the second ocular protrusion value for the second image is estimated as the ocular protrusion value on the day the image was acquired.

[0090] Below, a method and system for estimating an eye protrusion value according to one embodiment are described.

[0091]

[0092] 1. Eye protrusion value estimation system

[0093] The exophthalmos value is an indicator of how much the eyeball protrudes, and can be determined by the vertical distance between the corneal vertex and the outer rim of the orbit. In other words, the exophthalmos value is the vertical distance between the corneal vertex and the outer rim of the orbit, so it was difficult to estimate the exophthalmos value using only a frontal facial image.

[0094] However, according to the embodiments disclosed in the present application, the eye protrusion value can be estimated using a frontal facial image. A specific eye protrusion value estimation method is described below.

[0095] FIG. 1 is a diagram for explaining a process for estimating an eye protrusion value according to one embodiment.

[0096] Referring to FIG. 1, an image capturing module (110), a preprocessing module (120), a depth image generation module (130), a preprocessing module (140), and an eye protrusion value estimation module (150) can be used to estimate an eye protrusion value. Specific details for each module are described below.

[0097] (1) Image capturing module (110)

[0098] Referring to FIG. 1, the image capturing module (110) can generate a visible light image (115).

[0099] The image capturing module (110) may include a camera module and may include an optical lens, an image sensor, and an image processing unit.

[0100] An optical lens is a transparent optical device that focuses or disperses light through refraction, allowing the light to be transmitted to an image sensor. An image sensor is a device that converts an optical image into an electrical signal and may be composed of a chip integrating multiple photodiodes. For example, an image sensor may include a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). An image processing unit may process the captured image results to generate image information.

[0101] The image capturing module (110) may include a visible light camera module, and may generate a visible light image (115) through the visible light camera module. In this case, the visible light camera module refers to a camera module that detects visible light among light rays.

[0102] The image capturing module (110) can generate a visible light image (115) in a monocular manner. For example, a single camera module can be used to generate the visible light image (115). However, this is not limited thereto, and if the image capturing module (110) includes multiple camera modules, the image capturing module (110) can also generate a single visible light image (115) by synthesizing multiple visible light images acquired by each of the multiple camera modules in a monocular manner.

[0103] The visible light image (115) may be an image including two-dimensional pixels. Each pixel of the visible light image (115) may be assigned a value corresponding to the color and / or brightness of visible light detected by the camera module.

[0104] For example, each pixel of a visible light image (115) may be assigned a value corresponding to red, a value corresponding to green, a value corresponding to blue, and / or a value corresponding to brightness.

[0105] For a more specific example, each pixel of the visible light image (115) may be assigned a value between 0 and 255 corresponding to red, a value between 0 and 255 corresponding to green, a value between 0 and 255 corresponding to blue, and / or a value between 0 and 255 corresponding to brightness, but is not limited thereto.

[0106] When a subject is photographed through an image capturing module (110), the subject may appear in a visible light image (115).

[0107] For example, if a subject's face is captured using an image capturing module (110), the subject's face may appear in a visible light image (115). For another example, if a subject's eye is captured using an image capturing module (110), the subject's eye may appear in a visible light image (115).

[0108] (2) Preprocessing module (120)

[0109] Referring to FIG. 1, the preprocessing module (120) can preprocess a visible light image (115) to generate a preprocessed visible light image (125).

[0110] Preprocessing may include image cropping, image resizing, image flipping, image brightness adjustment, and / or image noise removal.

[0111] Image cropping refers to creating a cropped image by cropping a portion of an image. For example, image cropping may extract a portion of an image and create a new cropped image. Another example is image cropping, which may remove all but a portion of an image to create a cropped image. Image cropping is not limited to this, and may include preprocessing commonly understood as image cropping.

[0112] Image resizing refers to adjusting the number of pixels in an image. For example, horizontal resizing of an image may mean increasing or decreasing the number of horizontal pixels in the image, and vertical resizing of an image may mean increasing or decreasing the number of vertical pixels in the image. Image resizing is not limited to this, and may include preprocessing, which is commonly understood as image resizing.

[0113] Image flipping refers to exchanging the values ​​assigned to a specific pixel with the values ​​assigned to pixels in the opposite direction around a specific reference point. For example, the specific reference point may refer to a specific pixel location and / or a line connecting specific pixels. For example, the specific reference point may refer to the horizontal centerline and / or vertical centerline of the image. For a more specific example, image flipping horizontally may refer to mirroring the image left and right around the horizontal centerline of the image. Image flipping is not limited thereto, and may include preprocessing commonly understood as image flipping.

[0114] Image brightness adjustment refers to adjusting the values ​​assigned to pixels in an image to make it brighter or darker, either overall or in part. For example, image brightness adjustment can be performed through pixel value adjustment, histogram equalization, contrast adjustment, color channel adjustment, and / or binarization. Without being limited thereto, image brightness adjustment may include preprocessing commonly understood as image brightness adjustment.

[0115] Image denoising refers to reducing or removing noise from an image. For example, image denoising can be performed using a mean filter, a median filter, a Gaussian filter, and / or deep learning-based filters. However, image denoising is not limited to these methods, and may include preprocessing commonly understood as image denoising.

[0116] Although not limited thereto, image preprocessing may include various types of preprocessing that may be performed on an image before analyzing the image.

[0117] The preprocessed visible light image (125) is an image generated by preprocessing the visible light image (115). For example, the preprocessed visible light image (125) may be generated by cropping, resizing, inverting, adjusting brightness, and / or removing noise from the visible light image (115), but is not limited thereto.

[0118] (3) Depth image generation module (130)

[0119] Referring to FIG. 1, the depth image generation module (130) can generate a depth image (135) based on a visible light image (115).

[0120] The depth image generation module (130) may include a pre-learned depth map generation model.

[0121] The depth map generation model may be a model trained to estimate a depth value for at least one pixel included in an input image, assign the estimated depth value to a pixel of an output image, and ultimately generate a depth image corresponding to the input image. For example, the depth map generation model may be a model trained to estimate a depth value of an object represented by at least one pixel included in an input image, and assign the estimated depth value to a pixel of an output image, thereby ultimately generating a depth image corresponding to the input image.

[0122] The generated depth image may be an image comprising two-dimensional pixels. Each pixel of the depth image may be assigned an estimated depth value for an object represented by at least one pixel of the input image corresponding to each pixel. Alternatively, each pixel group of the depth image may be assigned an estimated depth value for an object represented by at least one pixel of the input image corresponding to each pixel group.

[0123] The depth value may be a relative value. For example, the depth value may be a value between a preset minimum value and a preset maximum value. The depth value may be a maximum value if the object represented by at least one pixel of the input image corresponding to the pixel of the depth image to which the depth value is assigned is the closest object in the input image. The depth value may be a minimum value if the object represented by at least one pixel of the input image corresponding to the pixel of the depth image to which the depth value is assigned is the farthest object in the input image. The depth value may be a value closer to the maximum value when the object represented by at least one pixel of the input image corresponding to the pixel of the depth image to which the depth value is assigned is a relatively closer object in the input image. The depth value may be a value closer to the minimum value when the object represented by at least one pixel of the input image corresponding to the pixel of the depth image to which the depth value is assigned is a relatively far object in the input image. Alternatively, the depth value may be a minimum value if the object represented by at least one pixel of the input image corresponding to the pixel of the depth image to which the depth value is assigned is the closest object in the input image. The depth value may be a maximum value if the object indicated by at least one pixel of the input image corresponding to the pixel of the depth image to which the depth value is assigned is the farthest object in the input image. The depth value may be a value closer to the minimum value when the object indicated by at least one pixel of the input image corresponding to the pixel of the depth image to which the depth value is assigned is an object located relatively close in the input image. The depth value may be a value closer to the maximum value when the object indicated by at least one pixel of the input image corresponding to the pixel of the depth image to which the depth value is assigned is an object located relatively far in the input image, but is not limited thereto.

[0124] The depth value can be an absolute value. For example, the depth value can be a value within a preset range. In this case, the depth value can be assigned an actual distance value between the camera and an object appearing in at least one pixel of the input image corresponding to the pixel of the depth image to which the depth value is assigned.

[0125] Meanwhile, the depth map generation model is not limited to the above-described examples, and may include various types of artificial neural network models that are commonly understood as artificial neural network models that generate depth maps. For example, the depth map generation model may be a Monocular Depth Estimation model that generates a depth map using a single image. Specifically, the depth map generation model may be a Monocular Depth Estimation with Image-Depth-Aware Self-Supervised Learning (MIDAS) model that estimates relative depth among Monocular Depth Estimation models. Alternatively, the depth map generation model may be, but is not limited to, a ZoeDepth model that estimates absolute depth among Monocular Depth Estimation models.

[0126] Each pixel of the depth image (135) may be assigned an estimated depth value based on the visible light image (115). For example, each pixel of the depth image (135) may be assigned an estimated depth value corresponding to at least one pixel included in the visible light image (115). As another example, a group of pixels of the depth image (135) may be assigned an estimated depth value corresponding to at least one pixel included in the visible light image (115), but is not limited thereto.

[0127] The size of the depth image (135) may be the same as the size of the visible light image (115). For example, the number of pixels of the depth image (135) may be the same as the number of pixels of the visible light image (115). However, this is not limited thereto, and the size of the depth image (135) may be smaller than the size of the visible light image (115), or the size of the depth image (135) may be larger than the size of the visible light image (115).

[0128] (4) Preprocessing module (140)

[0129] Referring to FIG. 1, the preprocessing module (140) can preprocess the depth image (135) to generate a preprocessed depth image (145).

[0130] Preprocessing may include image cropping, image resizing, image horizontal flipping, image brightness adjustment, and / or image noise removal. Specific details regarding preprocessing have been described above in the preprocessing module (120), so a duplicate description will be omitted.

[0131] The preprocessed depth image (145) is an image generated by preprocessing the depth image (135). For example, the preprocessed depth image (145) may be generated by cropping, resizing, inverting, adjusting brightness, and / or removing noise from the depth image (135), but is not limited thereto.

[0132] FIG. 2 is a drawing for explaining a visible light image (210) and a depth image (220) according to one embodiment.

[0133] Figure 2 (a) is a visible light image (210) generated by capturing the face of a target through the image capturing module (110) described above.

[0134] FIG. 2 (b) is a depth image (220) generated based on the visible light image (210) using the depth image generation module (130) described above. Referring to FIG. 2 (b), it can be confirmed that each pixel of the depth image (220) is assigned a depth value estimated based on the visible light image (210). Specifically, it can be confirmed that the closer an object represented by a pixel in the visible light image (210) is to an object, the corresponding pixel of the depth image (220) is displayed in a brighter color, and the farther an object represented by a pixel in the visible light image (210) is to an object, the corresponding pixel of the depth image (220) is displayed in a darker color.

[0135] Figure 2 (c) is a preprocessed visible light image (230) generated by preprocessing a visible light image (210) using the preprocessing module (120) described above. Referring to Figure 2 (c), the preprocessed visible light image (230) represents an image in which preprocessing is performed to crop a portion of the visible light image (210).

[0136] Figure 2 (d) is a preprocessed depth image (240) generated by preprocessing the depth image (220) using the preprocessing module (140) described above. Referring to Figure 2 (d), the preprocessed depth image (240) represents an image in which preprocessing is performed to crop a portion of the depth image (220).

[0137] For example, the preprocessed depth image (240) can be generated by cropping an area corresponding to the preprocessed visible light image (230) from the depth image (220). In this case, the area corresponding to the preprocessed visible light image may mean an area in the depth image that corresponds to an area in the visible light image where cropping is performed from the preprocessed visible light image. Specifically, the area corresponding to the preprocessed visible light image may mean a pixel position in the depth image that corresponds to a pixel position in the visible light image where cropping is performed from the preprocessed visible light image, but is not limited thereto.

[0138] For another example, a preprocessed depth image (240) can be generated based on a preprocessed visible light image (230) using a depth image generation module (130), and the method for generating a preprocessed depth image is not limited thereto.

[0139] (5) Ocular protrusion value estimation module (150)

[0140] Referring back to FIG. 1, the eye protrusion value estimation module (150) can output an estimated eye protrusion value (155) based on both the preprocessed visible light image (125) and the preprocessed depth image (145).

[0141] The eye protrusion value estimation module (150) may include a pre-learned eye protrusion value estimation model.

[0142] The eye protrusion value estimation model may be a model trained to input a visible light image and a depth image and output an estimated eye protrusion value for the eye of a target appearing in the visible light image.

[0143] For example, the eye protrusion value estimation model may be a model trained to receive a visible light image and a depth image as input and output an estimated eye protrusion value for the left eye of a subject appearing in the visible light image. As another example, the eye protrusion value estimation model may be a model trained to receive a visible light image and a depth image as input and output an estimated eye protrusion value for the right eye of a subject appearing in the visible light image. As yet another example, the eye protrusion value estimation model may be a model trained to receive a visible light image and a depth image as input and output an estimated eye protrusion value for the right eye and an estimated eye protrusion value for the left eye of a subject appearing in the visible light image.

[0144] An eye protrusion value estimation model can be trained using a visible light image, a depth image corresponding to the visible light image, and the eye protrusion value for the eye of the target appearing in the visible light image as training data. In this case, the depth image can be generated using a depth image generation module based on the visible light image.

[0145] For example, an eye protrusion value estimation model can be trained using a visible light image, a depth image corresponding to the visible light image, and an eye protrusion value for the left eye of a subject appearing in the visible light image as training data. For another example, an eye protrusion value estimation model can be trained using a visible light image, a depth image corresponding to the visible light image, and an eye protrusion value for the right eye of a subject appearing in the visible light image as training data. For yet another example, an eye protrusion value estimation model can be trained using a visible light image, a depth image corresponding to the visible light image, an eye protrusion value for the left eye of a subject appearing in the visible light image, and an eye protrusion value for the right eye of a subject appearing in the visible light image as training data.

[0146] Meanwhile, the eye protrusion value estimation model can be trained using the preprocessed visible light image, the preprocessed depth image, and the eye protrusion value for the target eye appearing in the preprocessed visible light image as training data.

[0147] In this case, the preprocessed depth image can be generated by generating a depth image through a depth image generation module based on a visible light image and preprocessing the generated depth image. That is, the preprocessed depth image can be generated by preprocessing a depth image corresponding to the visible light image. Alternatively, the preprocessed depth image can be generated through a depth image generation module based on the preprocessed visible light image, but is not limited thereto.

[0148] In order to increase the number of training data used for training the ocular protrusion value estimation model, a visible light image can be flipped left and right to generate a left-right flipped visible light image, and a depth image can be flipped left and right to generate a left-right flipped depth image.

[0149] For example, if the eye protrusion value estimation model is an eye protrusion value estimation model for the left eye, the eye protrusion value for the left eye of the subject appearing in the visible light image, the depth image, and the visible light image may be used as training data, and the left-right inverted visible light image, the left-right inverted depth image, and the eye protrusion value for the right eye of the subject appearing in the visible light image may be used as training data. As another example, if the eye protrusion value estimation model is an eye protrusion value estimation model for the right eye, the eye protrusion value for the right eye of the subject appearing in the visible light image, the depth image, and the visible light image may be used as training data, and the left-right inverted visible light image, the left-right inverted depth image, and the eye protrusion value for the left eye of the subject appearing in the visible light image may be used as training data. For another example, if the eye protrusion value estimation model is an eye protrusion value estimation model for both the right eye and the left eye, the visible light image, the depth image, the eye protrusion value for the right eye of the subject appearing in the visible light image, and the eye protrusion value for the left eye of the subject appearing in the visible light image can be used as training data, and the left-right inverted visible light image, the left-right inverted depth image, the eye protrusion value for the left eye of the subject appearing in the visible light image, and the eye protrusion value for the right eye of the subject appearing in the visible light image can be used as training data.

[0150] The ocular protrusion value estimation model may refer to a model learned using machine learning. Here, machine learning can be understood as a comprehensive concept that includes artificial neural networks and, further, deep learning. At least one of k-nearest neighbors, linear regression, logistic regression, support vector machine (SVM), decision tree, random forest, and neural network may be used as the algorithm for the ocular protrusion value estimation model. Here, at least one of an artificial neural network (ANN), a time delay neural network (TDNN), a deep neural network (DNN), a convolution neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM), and a residual neural network (ResNet) may be selected as the neural network.

[0151] FIG. 3 is a diagram for explaining the structure of an eye protrusion value estimation model according to one embodiment.

[0152] Referring to FIG. 3, the eye protrusion value estimation model may include a first layer (312), a second layer (322), and a third layer (332).

[0153] The first layer (312) can process a visible light image (311) and output a first intermediate result (313).

[0154] The second layer (322) can process the depth image (321) and output a second intermediate result (323).

[0155] The third intermediate result (331) can be output by concatenating the first intermediate result (313) and the second intermediate result (323).

[0156] The third layer (332) can process the third intermediate result (331) to output an eye protrusion value (333).

[0157] Although not illustrated in FIG. 3, the first layer (312) may include multiple layers. For example, the first layer (312) may include multiple convolutional layers. In another example, the first layer (312) may have a first ResNet structure, but is not limited thereto.

[0158] Although not illustrated in FIG. 3, the second layer (322) may include multiple layers. For example, the second layer (322) may include multiple convolutional layers. For another example, the second layer (322) may have a second ResNet structure. For another example, the second layer (322) may have a first ResNet structure, but is not limited thereto.

[0159] Although not illustrated in FIG. 3, the third layer (332) may include multiple layers. For example, the third layer (332) may include multiple convolutional layers. For another example, the third layer (332) may have a third ResNet structure, but is not limited thereto.

[0160] As described above, the eye protrusion value can be estimated using the image capturing module (110), the preprocessing module (120), the depth image generation module (!30), the preprocessing module (140), and the eye protrusion value estimation module (150). This is not limited thereto, and at least one module may be omitted.

[0161] (6) Variant example

[0162] FIG. 4 is a diagram for explaining a process for estimating an eye protrusion value according to one embodiment.

[0163] Referring to FIG. 4, an image capturing module (410), a depth image generation module (420), and an eye protrusion value estimation module (430) may be used to estimate an eye protrusion value. That is, a preprocessing module may not be used to estimate an eye protrusion value.

[0164] The image capturing module (410) can generate a visible light image (415). Specific details related to the image capturing module and the visible light image have been described above in (1) the image capturing module, so a duplicate description will be omitted.

[0165] The depth image generation module (420) can generate a depth image (425) based on a visible light image (415). Specific details related to the depth image generation module and the depth image have been described above in (3) the depth image generation module, so a duplicate description will be omitted.

[0166] The population protrusion value estimation module (430) can output an estimated eye protrusion value (435) based on both the visible light image (415) and the depth image (425). Specific details related to the eye protrusion value estimation module and the estimated eye protrusion value have been described above in (5) the eye protrusion value estimation module, so a duplicate description will be omitted.

[0167] FIG. 5 is a diagram for explaining a process for estimating an eye protrusion value according to one embodiment.

[0168] Referring to FIG. 5, an image capturing module (510), a preprocessing module (520), a depth image generation module (530), and an eye protrusion value estimation module (540) may be used to estimate an eye protrusion value. That is, a preprocessing module for a depth image (535) may not be used to estimate an eye protrusion value.

[0169] The image capturing module (510) can generate a visible light image (515). Specific details related to the image capturing module and the visible light image have been described above in (1) the image capturing module, so a duplicate description will be omitted.

[0170] The preprocessing module (520) can preprocess a visible light image (515) to generate a preprocessed visible light image (525). Since the specific details of the preprocessing have been described above in (2) the preprocessing module, a duplicate description will be omitted.

[0171] The depth image generation module (530) can generate a depth image (535) based on a preprocessed visible light image (525). Specific details related to the depth image generation module and the depth image have been described above in (3) the depth image generation module, so a duplicate description will be omitted.

[0172] The eye protrusion value estimation module (540) can output an estimated eye protrusion value (545) based on both the preprocessed visible light image (525) and the depth image (535). Specific details related to the eye protrusion value estimation module and the estimated eye protrusion value have been described above in (5) the eye protrusion value estimation module, so a duplicate description will be omitted.

[0173] (7) System

[0174] FIG. 6 is a diagram illustrating a system for estimating an eye protrusion value according to one embodiment.

[0175] A system for estimating an ocular protrusion value may include a user's mobile device (610) and a server (620).

[0176] A portable device (610) is a device that interacts directly and / or indirectly with a user.

[0177] The portable device (610) may include an image capturing module, a user interface, a communication device, memory, and a processor.

[0178] The image capturing module of the portable device (610) may include a camera module, and specific details related to the image capturing module have been described above in (1) the image capturing module, so a duplicate description is omitted.

[0179] The user interface of the portable device (610) can output various information according to the control command of the processor of the portable device (610). The user interface of the portable device (610) can include a display that visually outputs information to the user. The user interface of the portable device (610) can include a speaker that audibly outputs information to the user. The user interface of the portable device (610) can receive various information from the user. The user can input various information through the user interface of the portable device (610). The user interface of the portable device (610) can include input devices such as a keyboard, a mouse, and / or a touch screen.

[0180] The communication device of the portable device (610) can transmit and / or receive data and / or information from the outside via wired and / or wireless communication. The communication device can perform two-way or one-way communication.

[0181] The communication device of the portable device (610) may include a wireless communication module and / or a wired communication module. The wireless communication module may include a Wi-Fi communication module and / or a cellular communication module.

[0182] The memory of the portable device (610) can store various processing programs, parameters for performing program processing, or data resulting from such processing. For example, the memory of the portable device (610) can store instructions, algorithms, and / or executable codes for the operation of the processor of the portable device (610), which will be described later.

[0183] The memory of the portable device (610) can store visible light images captured by the image capture module. Specific details related to the visible light images have been described above in (1) the image capture module, so a duplicate description will be omitted.

[0184] The memory of the portable device (610) can store a preprocessed visible light image, a depth image, a preprocessed depth image, and / or an estimated eye protrusion value generated according to the operation of the processor of the portable device (610) described below. Specific details related to the preprocessed visible light image have been described above in (2) the preprocessing module, so a redundant description will be omitted. Specific details related to the depth image have been described above in (3) the depth image generation module, so a redundant description will be omitted. Specific details related to the preprocessed depth image have been described above in (4) the preprocessing module, so a redundant description will be omitted. Specific details related to the estimated eye protrusion value have been described above in (5) the eye protrusion value estimation module, so a redundant description will be omitted.

[0185] The memory of the portable device (610) may be implemented as a nonvolatile semiconductor memory, a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a flash memory, a random access memory (RAM), a read only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), or other types of tangible nonvolatile recording media.

[0186] The processor of the portable device (610) can control the overall operation of the portable device (610) and can operate according to instructions, algorithms, and / or executable codes stored in the memory of the portable device (610).

[0187] The processor of the portable device (610) can acquire a visible light image. For example, the portable device (610) can acquire a visible light image by capturing the visible light image using an image capturing module. Specific details related to the visible light image have been described above in (1) the image capturing module, so a duplicate description will be omitted. As another example, the portable device (610) can acquire a visible light image by receiving a visible light image stored in an external storage medium through an input / output interface without using the image capturing module.

[0188] The processor of the portable device (610) can perform the operations of the preprocessing module. Specifically, the processor of the portable device (610) can preprocess a visible light image to generate a preprocessed visible light image. Specific details related to the preprocessing module have been described above in (2) Preprocessing Module, so a duplicate description will be omitted.

[0189] The processor of the portable device (610) can perform the operation of the depth image generation module. Specifically, the processor of the portable device (610) can generate a depth image based on a visible light image. Specific details related to the depth image generation module have been described above in (3) Depth Image Generation Module, so a duplicate description will be omitted.

[0190] The processor of the portable device (610) can perform the operations of the preprocessing module. Specifically, the processor of the portable device (610) can preprocess a depth image to generate a preprocessed depth image. Specific details related to the preprocessing module have been described above in (4) Preprocessing Module, so a duplicate description will be omitted.

[0191] The processor of the portable device (610) can perform the operation of the eye protrusion value estimation module. Specifically, the processor of the portable device (610) can estimate the eye protrusion value based on both the preprocessed visible light image and the preprocessed depth image. Specific details related to the eye protrusion value estimation module have been described above in (5) the eye protrusion value estimation model, so a duplicate description will be omitted.

[0192] The processor of the portable device (610) may transmit data to and / or receive data from the server (620) via the network (630) using the communication device of the portable device (610). This is not limited thereto, and the processor of the portable device (610) may transmit and / or receive data directly to and from the server (620) using the communication device of the portable device (610).

[0193] The processor of the portable device (610) may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a state machine, an application specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), or a combination thereof.

[0194] The portable device (610) may include a smart phone, a tablet, a desktop, a laptop, and / or a digital camera.

[0195] The server (620) may include a communication device, memory, and a processor.

[0196] The communication device of the server (620) can transmit and / or receive data and / or information from the outside via wired and / or wireless communication. The communication device can perform two-way or one-way communication.

[0197] The communication device of the server (620) may include a wireless communication module and / or a wired communication module. The wireless communication module may include a Wi-Fi communication module and / or a cellular communication module.

[0198] The memory of the server (620) may store various processing programs, parameters for performing program processing, or data resulting from such processing. For example, the memory of the server (620) may store instructions, algorithms, and / or executable codes for the operation of the processor of the server (620), which will be described later.

[0199] The memory of the server (620) can store visible light images, preprocessed visible light images, depth images, preprocessed depth images, and estimated eye protrusion values ​​obtained from the portable device (610).

[0200] The memory of the server (620) may be implemented as a nonvolatile semiconductor memory, HDD, SSD, SDD, flash memory, RAM, ROM, EEPROM, or other types of nonvolatile recording media.

[0201] The processor of the server (620) can control the overall operation of the server (620) and can operate according to instructions, algorithms, and / or executable codes stored in the memory of the server (620).

[0202] The processor of the server (620) may transmit data to and / or receive data from the portable device (610) via the network (630) using the communication device of the server (620). This is not limited thereto, and the processor of the server (620) may transmit and / or receive data directly with the portable device (610) using the communication device of the server (620).

[0203] The processor of the server (620) may be implemented as a central processing unit, a graphics processing unit, a digital signal processing unit, a state machine, an application-specific semiconductor, a radio frequency integrated circuit, and a combination thereof.

[0204] Meanwhile, the operations of the above-described preprocessing module, depth image generation module, and eye protrusion value estimation module have been described as being performed in a portable device (610), but at least one of the preprocessing module, depth image generation module, and eye protrusion value estimation module may be performed in a server (620).

[0205] For example, when a preprocessed visible light image is generated using a preprocessing module in a portable device (610), a depth image is generated using a depth image generation module, a preprocessed depth image is generated using the preprocessing module, and the preprocessed visible light image and the preprocessed depth image are transmitted to a server (620), the server (620) can estimate an eye protrusion value using an eye protrusion value estimation module.

[0206] For another example, when a visible light image is transmitted from a mobile device (610) to a server (620), the server (620) can generate a preprocessed visible light image using a preprocessing module, generate a depth image using a depth image generation module, generate a preprocessed depth image using the preprocessing module, and estimate an eye protrusion value using an eye protrusion value estimation module.

[0207]

[0208] 2. Method for estimating eye protrusion values

[0209] Fig. 7 is a flowchart illustrating a process for estimating an eye protrusion value according to one embodiment.

[0210] Referring to FIG. 7, a method for estimating an eye protrusion value for a subject may include a step of acquiring a visible light image in which an eye of the subject appears (S710), a step of preprocessing the visible light image to generate a preprocessed visible light image (S720), a step of generating a depth image corresponding to the visible light image (S730), a step of preprocessing the depth image to generate a preprocessed depth image (S740), a step of applying both the preprocessed visible light image and the preprocessed depth image to an eye protrusion value estimation model to estimate an eye protrusion value for the eye of the subject (S750), and a step of providing information based on the estimated eye protrusion value (S760).

[0211] (1) Step of acquiring a visible light image (S710)

[0212] A visible light image of the subject can be obtained.

[0213] For example, a visible light image can be acquired through an image capturing module. Specific details regarding the image capturing module and the visible light image have been described above in 1. (1) Image capturing module of the ocular protrusion value estimation system, so a duplicate description will be omitted. For another example, the visible light image may be a visible light image acquired at any point in the past and stored in a portable device. For another example, the visible light image may be a visible light image received from an external device and stored in the portable device. The method for acquiring a visible light image is not limited to the above-described examples, and may be acquired in various ways.

[0214] At least a portion of the subject's face region may be displayed in the visible light image. The face region refers to an area including, but not limited to, the subject's eyes, eyebrows, forehead, nose, bridge of the nose, mouth, chin, and / or ears, and may refer to an area recognized as a face when the subject's face is viewed from the front. The full face region refers to an area including, but not limited to, the subject's eyes, eyebrows, forehead, nose, bridge of the nose, mouth, chin, and ears, and may refer to the entire area recognized as a face when the subject's face is viewed from the front. However, the full face region may not include the subject's ears depending on the shape of the subject's face, the distance between the subject's face and the image capturing module, etc.

[0215] For example, a visible light image may show one eye of a subject. For another example, a visible light image may show one eye and the bridge of the nose of a subject. For another example, a visible light image may show both eyes and the bridge of the nose of a subject. For another example, a visible light image may show one eye and one eyebrow of a subject. For another example, a visible light image may show one eye, one eyebrow, and the bridge of the nose of a subject. For another example, a visible light image may show both eyes, both eyebrows, and the bridge of the nose of a subject. For another example, a visible light image may show a region of a subject's face from above the chin to below the eyebrows. However, the visible light image may show the entire face including at least one eye of a subject.

[0216] A visible light image may be an image that satisfies the analysis conditions.

[0217] Analysis conditions refer to conditions of visible light images suitable for estimating eye protrusion values.

[0218] For example, the analysis conditions are at least one of: the subject's eyes appear in the visible light image; the subject's eyes appearing in the visible light image are located within a predetermined eye region; the subject's pupil appearing in the visible light image is located within a predetermined pupil region; the subject's face appearing in the visible light image is located within a predetermined face region; the face appearing in the visible light image faces the front; the left-right rotation angle of the face appearing in the visible light image is within a predetermined left-right angle range; the up-down rotation angle of the face appearing in the visible light image is within a predetermined up-down angle range; the left-right tilt of the face appearing in the visible light image is within a predetermined tilt angle range; the smile level of the face appearing in the visible light image is within a predetermined smile level; the subject's nose appears in the visible light image; the subject's mouth appears in the visible light image; the subject's chin appears in the visible light image; the brightness of the visible light image is within a predetermined brightness range; the contrast of the visible light image is within a predetermined contrast range; and the blur level of the visible light image is within a predetermined blur range. May include, but is not limited to:

[0219] Hereinafter, the "subject" refers to the person being photographed, and the "user" refers to the person operating the mobile device. While the subject and user are distinguished in the descriptions below, the subject and user may be the same person. In other words, the user may also photograph themselves.

[0220] A shooting guide may be provided to the user to obtain a visible light image that satisfies the analysis conditions.

[0221] For example, a shooting guide may be provided that allows a user to adjust the position and / or angle of a portable device while taking a picture of an object using the portable device. In another example, a shooting guide may be provided that allows a user to adjust the position and / or angle of a portable device after taking a picture of an object using the portable device. In another example, a shooting guide may be provided that allows a user to adjust the position and / or angle of a subject while taking a picture of an object using the portable device. In another example, a shooting guide may be provided that allows a user to adjust the position and / or angle of a subject after taking a picture of an object using the portable device. In another example, a shooting guide may be provided that allows a user to adjust the brightness of the surrounding environment and / or the brightness of lighting while taking a picture of an object using the portable device. In another example, a shooting guide may be provided that allows a user to adjust the brightness of the surrounding environment and / or the brightness of lighting after taking a picture of an object using the portable device.

[0222] For example, the shooting guide may be provided in the form of guidelines along with a preview image on the display of the mobile device. For another specific example, the shooting guide may be provided in the form of text on the display of the mobile device. For another specific example, the shooting guide may be provided in the form of voice through the speaker of the mobile device.

[0223] The provision of shooting guides is not limited to the examples described above, and may be provided in various forms to guide the user and / or subject so that the visible light image satisfies the analysis conditions.

[0224] Figure 8 is a drawing for explaining a shooting guide according to one embodiment.

[0225] Referring to FIG. 8, a first guide (811), a second guide (812), a third guide (813), and a fourth guide (814) may be provided along with a preview image (810) through the display of the portable device.

[0226] The first guide (811) may be output to indicate the appropriate position of the subject's right eye. The second guide (812) may be output to indicate the appropriate position of the subject's left eye. The third guide (813) may be output to indicate the appropriate occupancy ratio of the subject's face. The fourth guide (814) may be output to indicate the appropriate left-right rotation angle of the subject's face.

[0227] The first guide (811) may be output to directly or indirectly indicate the proper position of the subject's right eye. For example, the first guide (811) may be output at the point where the subject's right eye should be positioned, thereby helping the user align the subject's right eye with the first guide (811). The first guide (811) may have a shape that indicates the outline of the iris area when the subject's eye is in the proper position, as illustrated in FIG. 8. Alternatively, the first guide (811) may have a shape that indicates the position of the pupil when the subject's eye is in the proper position, or an outline of the subject's eye.

[0228] The second guide (812) may be output to directly or indirectly indicate the proper position of the subject's left eye. For example, the second guide (812) may be output at the point where the subject's left eye should be positioned to help the user align the subject's left eye with the second guide (812). The second guide (812) may have a shape that indicates the outline of the iris area when the subject's eye is in the proper position, as illustrated in FIG. 8. Alternatively, the second guide (812) may have a shape that indicates the position of the pupil when the subject's eye is in the proper position, or an outline of the subject's eye.

[0229] The third guide (813) can be output to directly or indirectly indicate the appropriate proportion (portion) of the subject's face in the image. The third guide (813) can indicate the area where the subject's face should be located in a circular shape, or indicate the distance between the subject's face and the mobile device, thereby guiding the capture of an image in which the subject's face appears in the appropriate proportion in the image.

[0230] The fourth guide (814) can be output to directly or indirectly display the left-right rotation angle of the subject's face. The fourth guide (814) can display the left-right rotation angle of the face numerically, but can also display it as a vertical line extending in the vertical direction of the display. In this case, the vertical line can be a vertical line passing through the area where the nose of the face should be located when the subject's two eyes correspond to the first guide (811) and the second guide (812) respectively and the left-right rotation angle of the face is 0.

[0231] The first guide (811) and the second guide (812) may be provided symmetrically on the display. The first guide (811) and the second guide (812) may be provided symmetrically with respect to the fourth guide (814).

[0232] FIG. 9 is a drawing for explaining a shooting guide according to one embodiment.

[0233] Referring to (a) of FIG. 9, a fourth guide (911) and a fifth guide (912) may be provided together with a preview image (910).

[0234] The fourth guide (911) is a fixed shooting guide and can be provided regardless of the state of the preview image. According to Fig. 9 (a), the fourth guide (911) for indicating the left and right rotation angle of the subject's face can be provided in the form of a vertical line.

[0235] The fifth guide (912) is a real-time shooting guide, and can be generated based on information obtained by analyzing a preview image. According to (a) of Fig. 9, the preview image (910) is analyzed to calculate the left-right rotation angle of the face, and the fifth guide (912) corresponding to the calculated left-right rotation angle of the face can be provided.

[0236] The real-time shooting guide may be provided in a shape corresponding to the fixed shooting guide. For example, if the fixed shooting guide is in the shape of a vertical line, the real-time shooting guide corresponding to the fixed shooting guide may also be in the shape of a vertical line. For another example, if the fixed shooting guide is in the shape of a circle, the real-time shooting guide corresponding to the fixed shooting guide may also be in the shape of a circle.

[0237] According to (a) of FIG. 9, since the left-right rotation angle of the face in the preview image (910) does not satisfy the condition, guidance may be provided to satisfy the condition. For example, the phrase "Move your face left-right to align the vertical center line with the center line" may be displayed on the display, or a guidance voice may be provided through the speaker.

[0238] Figure 9 (b) illustrates a state in which the left and right rotation angles of the subject's face satisfy the condition. As illustrated in Figure 9 (b), if the fourth guide and fifth guide (921) provided together with the preview image (920) match each other, it can be determined that the condition is satisfied.

[0239] Although only vertical lines are shown in Figure 9 to guide the left and right rotation angles of the subject's face using fixed and real-time shooting guides, this is not a limitation. Depending on the desired shooting guide, an appropriate form of fixed and real-time shooting guide may be provided.

[0240] FIG. 10 is a drawing for explaining a shooting guide according to one embodiment.

[0241] Referring to FIG. 10, a third guide (1011), a fourth guide (1012), a sixth guide (1013), and an icon guide (1014) may be provided along with a preview image (1010) through the display of the mobile device.

[0242] The third guide (1011) can be output to directly or indirectly indicate the appropriate proportion (portion) of the subject's face in the image. The third guide (1011) can indicate the area where the subject's face should be located in the form of a circle, an ellipse, and / or a face shape, or indicate the distance between the subject's face and the mobile device, thereby guiding the capture of an image in which the subject's face appears in the appropriate proportion in the image.

[0243] The third guide (1011) may be displayed as a shape of a face including the outline of the ear. Compared to an oval shape without an outline of the ear, the third guide (1011) with the outline of the ear expressed together is displayed, thereby guiding the subject to look straight ahead, making it easier to obtain an image that satisfies the condition that the up-down rotation angle and the left-right rotation angle of the subject's face are 0.

[0244] The fourth guide (1012) can be output to directly or indirectly display the left-right rotation angle of the subject's face. The fourth guide (1012) can display the left-right rotation angle of the face numerically, but can also display it as a vertical line extending in the vertical direction of the display. In this case, the vertical line can be a vertical line passing through the area where the nose of the face should be located when the left-right rotation angle of the face is 0.

[0245] The sixth guide (1013) can be output to directly or indirectly display the vertical rotation angle of the subject's face. The sixth guide (1013) can display the vertical rotation angle of the face numerically, but can also display it as a horizontal line extending in the horizontal direction of the display. In this case, the horizontal line can be a horizontal line passing through the area where the nose of the face should be located when the vertical rotation angle of the face is 0.

[0246] Referring to FIG. 10, an icon guide (1014) may be provided to indicate whether a face in a preview image (1010) satisfies an analysis condition. For example, as illustrated in FIG. 10, if the left-right rotation angle of the face satisfies the condition, the icon guide (1014) may indicate that the left-right rotation angle of the face satisfies the condition, and if the proportion of the face in the image is not appropriate, the icon guide (1014) may indicate that the appropriate distance of the face does not satisfy the condition. Through the display of the icon guide (1014), a user and / or a subject can easily determine whether the analysis condition is satisfied.

[0247] One visible light image can be acquired.

[0248] Alternatively, two or more visible light images can be acquired to consistently estimate accurate ocular protrusion values. For example, three visible light images can be acquired. As will be described later, if two or more visible light images are acquired, the ocular protrusion value can be estimated for each visible light image, and then the average of the estimated values ​​can be estimated as the final ocular protrusion value. Estimating ocular protrusion value using the average value of multiple visible light images in this way reduces the error compared to using only one visible light image, allowing for more accurate ocular protrusion value estimation. In addition, although it is most important to implement a system to have high accuracy in estimating ocular protrusion values, it is realistically impossible to have a 100% accuracy rate. Therefore, a system with a constant difference from the correct answer or a constant direction of difference from the correct answer can be evaluated as a better system. Accordingly, the applicant 'acquired multiple photographs per day and analyzed each of them' to extract the ocular protrusion value corresponding to each day as their representative value, and estimated this at daily or weekly intervals, thereby ensuring a relatively high consistency in ocular protrusion values ​​between days or weeks compared to analyzing one photograph per day. The related experiment is described in Experimental Example 8 below.

[0249] Once visible light images are acquired, steps may be performed to estimate eye protrusion values ​​for each acquired visible light image. Specifically, steps S720, S730, S740, and S750, described below, may be performed for each acquired visible light image.

[0250] Below, steps S720, S730, S740 and S750 are described in detail.

[0251] (2) Step of preprocessing a visible light image to generate a preprocessed visible light image (S720)

[0252] The acquired visible light image can be preprocessed to generate a preprocessed visible light image. For example, the visible light image can be preprocessed using a preprocessing module. Specific details regarding the preprocessing module and the preprocessed visible light image have been described above in (2) Preprocessing Module of 1. Eye Protrusion Value Estimation System, so a duplicate description will be omitted.

[0253] Preprocessing can be performed on a visible light image showing the subject's face and background to generate a preprocessed visible light image in which the subject's face is significantly represented. Alternatively, preprocessing can be performed on a visible light image showing the subject's entire face to generate a preprocessed visible light image in which a portion of the subject's face is represented.

[0254] For example, a visible light image showing the entire face of a subject may be cropped to generate a preprocessed visible light image showing the facial area between under the eyebrows and above the chin of the subject. In another example, a visible light image showing the entire face of a subject may be cropped to generate a preprocessed visible light image showing the facial area between under the eyebrows and above the tip of the nose of the subject. In another example, a visible light image showing the entire face of a subject may be cropped to generate a preprocessed visible light image showing the eye areas of both subjects and the bridge of the nose of the subject. In another example, a visible light image showing the entire face of a subject may be cropped to generate a preprocessed visible light image showing one eye area of ​​the subject and the bridge of the nose of the subject. In another example, a visible light image showing the entire face of a subject may be cropped to generate a preprocessed visible light image showing one eye area of ​​the subject. The cropping method for the visible light image is not limited to the above-described examples, and it is obvious that the image can be cropped to include an appropriate face region for estimating the eye protrusion value.

[0255] Meanwhile, estimating eye protrusion values ​​using images that include both eyes and / or the bridge of the nose can further improve the accuracy of the estimated eye protrusion values. For example, preprocessing visible-light images to include both eyes and / or the bridge of the nose can further improve the accuracy of estimating eye protrusion values.

[0256] The reason is that, in order to accurately analyze distance or perspective, such as relative depth values, the image to be analyzed should have as wide an area as possible for more accurate analysis. In addition, since exophthalmos often causes only one eye to protrude and the left and right areas of the face are similar, the accuracy of estimating the exophthalmos value can be improved if the preprocessed visible light image includes both eyes. In addition, the accuracy of estimating the exophthalmos value can be improved by including the bridge of the nose, which can be used as a comparison target when estimating the relative depth value of the eye, including the bridge of the nose. The experimental results related to this are described in Experimental Examples 1 to 5 in Experimental Examples of the 4. Exophthalmos Value Estimation Model described below.

[0257] Fig. 11 is a drawing for explaining a preprocessing method of a visible light image according to one embodiment.

[0258] Referring to FIG. 11, a visible light image (1110) may show a facial region of a subject. Specifically, the entire facial region including the subject's right eye (1111), the subject's left eye (1112), and the bridge of the subject's nose (1113) may be shown.

[0259] As illustrated in FIG. 11, a visible light image (1110) can be preprocessed to generate a preprocessed visible light image (1120). Specifically, the visible light image (1110) can be cropped to generate a preprocessed visible light image (1120).

[0260] The preprocessed visible light image (1120) may show a portion of the face including the subject's right eye (1121), the subject's left eye (1122), and the subject's nose bridge (1123).

[0261] That is, according to one embodiment, a visible light image (1110) showing the entire face area can be preprocessed to generate a visible light image (1120) showing a part of the face area.

[0262] FIG. 12 is a drawing for explaining a preprocessing method of a visible light image according to one embodiment.

[0263] Referring to FIG. 12, the visible light image (1210) may show the face area of ​​a subject. Specifically, the entire face area including the subject's right eye (1211), the subject's left eye (1212), and the bridge of the subject's nose (1213) may be shown.

[0264] As illustrated in FIG. 12, a visible light image (1210) can be preprocessed to generate a preprocessed visible light image (1220). Specifically, the visible light image (1210) can be cropped to generate a preprocessed visible light image (1220).

[0265] The preprocessed visible light image (1220) may show a portion of the face including the subject's right eye (1221) and the bridge of the subject's nose (1223).

[0266] That is, according to one embodiment, a visible light image (1210) showing the entire face area can be preprocessed to generate a visible light image (1220) showing the subject's right eye (1221) and the bridge of the subject's nose (1223).

[0267] In Fig. 12, preprocessing is performed so that the subject's right eye (1221) and the subject's nose bridge (1223) appear in the preprocessed visible light image (1220), but this is not limited thereto, and preprocessing may be performed on the visible light image so that the subject's left eye and the subject's nose bridge appear in the preprocessed visible light image.

[0268] Fig. 13 is a drawing for explaining a preprocessing method of a visible light image according to one embodiment.

[0269] Referring to FIG. 13, the visible light image (1310) may show the facial region of a subject. Specifically, the entire facial region including the subject's right eye (1311), the subject's left eye (1312), and the bridge of the subject's nose (1313) may be shown.

[0270] As illustrated in FIG. 13, a visible light image (1310) can be preprocessed to generate a preprocessed visible light image (1320). Specifically, the visible light image (1310) can be cropped to generate a preprocessed visible light image (1320).

[0271] The preprocessed visible light image (1320) may show a portion of the face including the subject's right eye (1321).

[0272] That is, according to one embodiment, a visible light image (1310) showing the entire face area can be preprocessed to generate a visible light image (1320) showing the subject's right eye (1321).

[0273] In Fig. 13, preprocessing is performed so that the right eye (1321) of the subject appears in the preprocessed visible light image (1320), but this is not limited thereto, and preprocessing of the visible light image may also be performed so that the left eye of the subject appears in the preprocessed visible light image.

[0274] FIG. 14 is a drawing for explaining a preprocessing method of a visible light image according to one embodiment.

[0275] Referring to FIG. 14, the visible light image (1410) may show the face area of ​​the subject. Specifically, the entire face area including the subject's right eye (1411), the subject's left eye (1412), and the bridge of the subject's nose (1413) may be shown.

[0276] As illustrated in FIG. 14, a visible light image (1410) can be preprocessed to generate a preprocessed visible light image (1420). Specifically, the visible light image (1410) can be cropped to generate a preprocessed visible light image (1420).

[0277] The preprocessed visible light image (1420) may show a portion of the face including the subject's right eye (1421) and the subject's left eye (1422).

[0278] That is, according to one embodiment, a visible light image (1410) showing the entire face area can be preprocessed to generate a visible light image (1420) showing the subject's right eye (1421) and the subject's left eye (1422).

[0279] (3) Step of generating a depth image corresponding to a visible light image (S730)

[0280] Depth images can be generated based on visible light images.

[0281] For example, a depth image can be generated from a visible light image using a depth image generation module. The depth image generation module may include a pre-trained depth map generation model. Specific details related to the depth image generation module, the depth map generation model, and the depth image have been described above in 1. (3) Depth Image Generation Module of the Eye Protrusion Value Estimation System, so a duplicate description will be omitted.

[0282] When generating a depth image, it may be preferable to use a visible light image that shows the entire face rather than a visible light image that shows only a portion of the face. Depth images represent the relative depth values ​​of objects appearing in the underlying visible light image, so using a visible light image that shows the entire face can better reflect the perspective of objects appearing in the visible light image.

[0283] When generating a depth image, it may be preferable to use a visible light image in which the proportion of the face region is high rather than a visible light image in which the proportion of the background region is high. This is because a depth image represents the relative depth value of an object appearing in the underlying visible light image, and if the proportion of the background region is high, the depth value for the face region may not be properly reflected.

[0284] In order to generate a depth image using a visible light image with a high proportion of the entire facial area, a visible light image with a high proportion of the entire facial area may be acquired in the step of acquiring the visible light image. Alternatively, before generating a depth image based on the acquired visible light image, the visible light image may be cropped to generate a visible light image with a high proportion of the entire facial area, and the depth image may be generated based on the cropped visible light image.

[0285] (4) Step of preprocessing the depth image to generate a preprocessed depth image (S740)

[0286] The generated depth image can be preprocessed to generate a preprocessed depth image. For example, the depth image can be preprocessed using a preprocessing module. Specific details regarding the preprocessing module and the preprocessed depth image have been described above in (4) Preprocessing Module of 1. Eye Protrusion Value Estimation System, so a duplicate description will be omitted.

[0287] (5) Step (S750) of estimating the eye protrusion value for the subject's eye by applying both the preprocessed visible light image and the preprocessed depth image to the eye protrusion value estimation model.

[0288] By applying both the preprocessed visible light image and the preprocessed depth image to the ocular protrusion value estimation model, the ocular protrusion value of the subject's eyes can be estimated. The specific details related to the ocular protrusion value estimation model and ocular protrusion value estimation have been described above in 1. (5) Ocular Protrusion Value Estimation Module of the Ocular Protrusion Value Estimation System, so a duplicate description will be omitted.

[0289] The exophthalmos value can be estimated for each visible light image.

[0290] For example, if one visible light image is acquired as described above, one eye protrusion value can be estimated for one visible light image.

[0291] For another example, if multiple visible light images are acquired as described above, multiple eye protrusion values ​​can be estimated based on each visible light image, and the average of these values ​​can be estimated as the final eye protrusion value. Estimating the average value as the eye protrusion value in this way allows for consistently more accurate eye protrusion values ​​to be estimated compared to eye protrusion values ​​estimated based on a single visible light image.

[0292] (6) Step for providing information based on the estimated eye protrusion value (S760)

[0293] Information can be provided based on the estimated eye protrusion values.

[0294] For example, the eye protrusion value shown in the visible light image can be provided based on the estimated eye protrusion value.

[0295] FIG. 15 is a drawing for explaining a UI that provides an eye protrusion value according to one embodiment.

[0296] The ocular protrusion values ​​may be provided numerically for each eye of the subject. For example, as illustrated in (a) of FIG. 15, the estimated ocular protrusion value for the subject's left eye may be 17.9 mm, and the estimated ocular protrusion value for the subject's right eye may be 17.7 mm.

[0297] At this time, as shown in (b) of Fig. 15, the eye protrusion values ​​for each of the subject's two eyes can be provided together with the visible light image that serves as the basis for the analysis.

[0298] FIG. 16 is a drawing for explaining a UI that provides a graph of eye protrusion values ​​according to one embodiment.

[0299] Referring to FIG. 16, a UI that provides a graph of eye protrusion values ​​may display a graph interface (1610, 1620), an eye position indicator (1611, 1621), a date indicator (1612, 1622), a number indicator (1613, 1623), a bar indicator (1614, 1624), and a comparison indicator (1615, 1625).

[0300] The graph interface (1610, 1620) can provide estimated eye protrusion values.

[0301] The graph interface (1610, 1620) can provide a graph of estimated eye protrusion values ​​on a daily and / or weekly basis.

[0302] For example, the graph interface (1610, 1620) may provide estimated eye protrusion values ​​on a daily and / or weekly basis as a bar graph. For example, as illustrated in FIG. 16, eye protrusion values ​​may be estimated daily, and the estimated eye protrusion values ​​may be provided in a graph format. This has the advantage of allowing the patient to check eye protrusion values ​​more consistently and frequently, as the eye protrusion values ​​can be determined at a shorter interval and with greater frequency than if the patient were to visit the hospital in person.

[0303] The eye position indicators (1611, 1621) may indicate the eye positions of the subject's two eyes, which are provided by the graph interfaces (1610, 1620). For example, as illustrated in FIG. 16, if the eye position indicator (1611) is displayed to the left, the graph interface (1610) may provide an estimated eye protrusion value for the subject's left eye. For another example, if the eye position indicator (1622) is displayed to the right, the graph interface (1620) may provide an estimated eye protrusion value for the subject's right eye.

[0304] The date indicator (1612, 1622) may display the date on which the eye protrusion value was estimated.

[0305] The numeric indicators (1613, 1623) may display the estimated eye protrusion values ​​as specific numbers. The numeric indicators (1613, 1623) may be implemented to display the eye protrusion values ​​corresponding to the corresponding dates among the dates indicated in the date indicators (1612, 1622).

[0306] The bar indicators (1614, 1624) may display the estimated eye protrusion values ​​as bars. The bar indicators (1614, 1624) may be implemented to display the eye protrusion values ​​corresponding to the corresponding dates among the dates indicated in the date indicators (1612, 1622).

[0307] Referring to FIG. 16, the estimated eye protrusion value may be provided as a specific numerical value and / or a bar corresponding to the numerical value, together with the date on which the eye protrusion value was estimated. For example, the estimated eye protrusion value may be implemented as a date indicator (1612, 1622), and the eye protrusion value corresponding to the date indicator (1612, 1622) may be implemented as a numerical indicator (1613, 1623) and / or a bar indicator (1614, 1624). For example, if the exophthalmos value of the left eye was measured as 17.9 mm on January 6, 2025, the eye position indicator (1611) on the graph interface (1610) may be displayed to the left, the date indicator (1612) may be displayed as 25.01.06, the number indicator (1613) may be displayed as 17.9, and the bar indicator (1614) may be displayed as illustrated in FIG. 16.

[0308] As another example, although not illustrated separately, information displayed on the graph interface (1610, 1620) may be provided in the form of a line graph, indicating estimated eye protrusion values ​​on a daily and / or weekly basis. In this case, the eye protrusion values ​​may be displayed as spots rather than as bar indicators (1614, 1624) illustrated in FIG. 16, and the graph interface (1610, 1620) may provide the eye protrusion values ​​in the form of a line graph connecting the points with lines. The eye protrusion values ​​measured on a daily or weekly basis may be provided together with a comparison indicator (1615, 1625).

[0309] For another example, if the estimated eye protrusion value exceeds a preset threshold, guidance information for visiting the subject may be provided. In this case, the threshold may be determined based on factors such as the subject's race and / or facial shape.

[0310] For another example, if the estimated eye protrusion value increases by a preset threshold compared to the subject's past eye protrusion values, guidance information for visiting the subject may be provided. In this case, the threshold may be determined based on factors such as the subject's race and / or facial shape. The subject's past eye protrusion values ​​may be stored on the mobile device. Alternatively, the eye protrusion values ​​may be estimated based on visible light images of the subject's past face.

[0311] As another example, information can be provided that assesses the activity of thyroid ophthalmopathy in a subject based on the estimated exophthalmos value. Specifically, the estimated exophthalmos value can be used to determine a Clinical Activity Score (CAS) associated with thyroid ophthalmopathy activity. For example, if the estimated exophthalmos value increases by 2 mm or more, a CAS score of 1 point can be assigned.

[0312] As another example, information can be provided that assesses the severity of thyroid ophthalmopathy in a subject based on the estimated exophthalmos value. Specifically, the estimated exophthalmos value can be used to assess the severity of thyroid ophthalmopathy. For example, if the estimated exophthalmos value increases by 2 mm or more, the severity of thyroid ophthalmopathy can be determined to be high.

[0313] As another example, the estimated eye protrusion values ​​may provide auxiliary information for medical personnel's diagnosis and / or examination.

[0314] As another example, information may be provided to determine whether drug treatment is necessary for a subject based on the estimated eye extrusion value.

[0315] As another example, information can be provided to determine whether surgery is needed for a subject based on the estimated eye protrusion value.

[0316] As another example, information can be provided to determine the extent of surgery for a subject based on the estimated eye protrusion value. (Past images can be used to estimate eye protrusion values ​​at past points in time.)

[0317] As another example, eye protrusion monitoring information for a subject can be provided based on the estimated eye protrusion value. For example, eye protrusion values ​​can be estimated on a daily or weekly basis, and the estimated eye protrusion values ​​on a daily or weekly basis can be provided in a graph format. Furthermore, the monitoring information described above can be utilized as an auxiliary means to verify the effectiveness of a treatment for exophthalmos in clinical trials.

[0318] (7) Variant example

[0319] In the process of estimating the eye protrusion value according to an embodiment described using Fig. 7, some steps may be omitted.

[0320] For example, the visible light image capturing step can be omitted. In this case, the visible light image can be received externally. Alternatively, a visible light image stored on a portable device can be used.

[0321] For example, the preprocessing step can be omitted. In this case, the acquired visible light image itself can be utilized.

[0322] For example, the depth image generation step can be omitted. In this case, the depth image can be received externally. Alternatively, a depth image stored on a portable device can be used.

[0323] For example, the step of providing information can be omitted. In this case, the estimated eye protrusion value can only be stored on the portable device.

[0324] Fig. 17 is a flowchart illustrating a process for estimating an eye protrusion value according to one embodiment.

[0325] Referring to FIG. 17, a method for estimating an eye protrusion value for a subject may include a step of acquiring a visible light image in which an eye of the subject appears (S1710), a step of generating a depth image corresponding to the visible light image (S1720), a step of applying both the visible light image and the depth image to an eye protrusion value estimation model to estimate the eye protrusion value (S1730), and a step of providing information based on the estimated eye protrusion value (S1740). That is, a preprocessing step may not be used to estimate the eye protrusion value.

[0326] At this time, one or more visible light images can be acquired in step S1710, and if there are two or more acquired visible light images, steps S1720 and S1730 can be performed for each visible light image to estimate an eye protrusion value, and an average value of the eye protrusion values ​​thus acquired can be estimated as a final eye protrusion value. FIG. 18 is a flowchart for explaining a process of estimating an eye protrusion value according to one embodiment.

[0327] Referring to FIG. 18, a method for estimating an eye protrusion value for a target may include a step of acquiring a visible light image in which an eye of the target appears (S1810), a step of preprocessing the visible light image to generate a preprocessed visible light image (S1820), a step of generating a depth image corresponding to the preprocessed visible light image (S1830), a step of applying both the preprocessed visible light image and the depth image to an eye protrusion value estimation model to estimate the eye protrusion value (S1840), and a step of providing information based on the estimated eye protrusion value (S1850). That is, a preprocessing step for the depth image may not be used to estimate the eye protrusion value.

[0328] At this time, one or more visible light images can be acquired in step S1810, and if there are two or more visible light images acquired, steps S1820, S1830, and S1840 can be performed for each visible light image to estimate the eye protrusion value, and the average value of the eye protrusion values ​​acquired in this way can be estimated as the final eye protrusion value.

[0329]

[0330] 3. Comparative example of methods for estimating eye protrusion values

[0331] (1) Comparative Example 1

[0332] For each of the 1,136 visible light images showing the subject's facial region, a 3D facial landmark detection model was applied to calculate the z-axis value between the center of the pupil and the tail of the eye.

[0333] In this case, MAE (mm) and Pearson Correlation were as follows.

[0334] MAE(mm)Pearson CorrelationLeft eye5.35-0.01Right eye5.42-0.02

[0335] (2) Second comparative example

[0336] For each of the 1,136 visible light images showing the subject's face area and the corresponding 1,136 depth images, the correlation with the eye protrusion value was calculated based on the difference between the pixel value at the center of the pupil and the pixel value at the tail of the eye.

[0337] In this case, the Pearson Correlation was as follows.

[0338] Pearson Correlation Left Eye 0.050475 Right Eye 0.035497

[0339] (3) Third comparative example

[0340] Using 1,136 visible light images showing both eyes and the bridge of the nose of the subject (see Fig. 11) and the eye protrusion value of one eye corresponding to each of the 1,136 visible light images as a data set, an eye protrusion value estimation model was trained using an artificial neural network.

[0341] In this case, visible light images were used as input data and eye protrusion values ​​were used as label values.

[0342] In this case, the 5-fold cross validation method was used.

[0343] In this case, Pearson Correlation, R 2 , MAE (mm) and MAPE (%) were as follows.

[0344] Pearson CorrelationR 2 MAE(mm)MAPE(%)Left eye0.780.611.417.77Right eye0.770.591.407.84

[0345] 4. Experimental example for the eye protrusion value estimation model

[0346] (1) Experimental example 1

[0347] An eye protrusion value estimation model was trained using an artificial neural network using 1,136 visible light images (see Fig. 11) showing both eyes and the bridge of the nose of the subject, 1,136 depth images corresponding to each of the 1,136 visible light images, and the eye protrusion value of one eye corresponding to each of the 1,136 visible light images as a data set.

[0348] In this case, visible light images and depth images were used as input data, and eye protrusion values ​​were used as label values.

[0349] In this case, the MIDAS model was used as the depth map generation model.

[0350] In this case, the 5-fold cross validation method was used.

[0351] In this case, Pearson Correlation, R 2 , MAE (mm) and MAPE (%) were as follows.

[0352] Pearson CorrelationR 2 MAE(mm)MAPE(%)Left eye0.800.631.347.43Right eye0.800.651.317.37

[0353] Through the above results, we were able to confirm that the ocular protrusion value can be accurately estimated using a depth image based on a 2D image such as a visible light image.

[0354] (2) Experimental example 2

[0355] An eye protrusion value estimation model was trained using an artificial neural network using 1,136 visible light images (see Fig. 11) showing both eyes and the bridge of the nose of the subject, 1,136 depth images corresponding to each of the 1,136 visible light images, and the eye protrusion value of one eye corresponding to each of the 1,136 visible light images as a data set.

[0356] In this case, visible light images and depth images were used as input data, and eye protrusion values ​​were used as label values.

[0357] In this case, the ZoeDepth model was used as the depth map generation model.

[0358] In this case, the 5-fold cross validation method was used.

[0359] In this case, Pearson Correlation, R 2 , MAE (mm) and MAPE (%) were as follows.

[0360] Pearson CorrelationR 2 MAE(mm)MAPE(%)Left eye0.820.671.287.11Right eye0.810.661.297.24

[0361] The first experimental example used the MIDAS model as a depth map generation model, and the second experimental example used the ZoeDepth model as a depth map generation model. Therefore, the first and second experimental examples used different types of depth map generation models to generate depth images to estimate the eye protrusion value. Comparing the results of the first and second experimental examples, it was confirmed that there was no significant difference in the performance of estimating the eye protrusion value, and both experimental examples could estimate the eye protrusion value with sufficient accuracy.

[0362] Through this, it was confirmed that the ocular protrusion value can be accurately estimated using a depth image based on a 2D image such as a visible light image, even if the type of depth map generation model that generates the depth image is different.

[0363]

[0364] (3) Experimental example 3

[0365] An eye protrusion value estimation model was trained using an artificial neural network using 1,136 visible light images (see Fig. 12) showing one eye and the bridge of the nose of the subject, 1,136 depth images corresponding to each of the 1,136 visible light images, and the eye protrusion value of one eye corresponding to each of the 1,136 visible light images as a data set.

[0366] In this case, visible light images and depth images were used as input data, and eye protrusion values ​​were used as label values.

[0367] In this case, the 5-fold cross validation method was used.

[0368] In this case, Pearson Correlation and MAE (mm) were as follows.

[0369] Pearson Correlation MAE (mm) Left eye 0.80 1.34 Right eye 0.80 1.33

[0370] Comparing the results of the third experimental example with the results of the first and second experimental examples, it was confirmed that estimating the eye protrusion value using an image including both eyes of the subject rather than one eye showed higher accuracy.

[0371] (4) Experimental example 4

[0372] An eye protrusion value estimation model was trained using an artificial neural network using 1,136 visible light images (see Fig. 13) showing one eye of the subject, 1,136 depth images corresponding to each of the 1,136 visible light images, and the eye protrusion value of one eye corresponding to each of the 1,136 visible light images as a data set.

[0373] In this case, visible light images and depth images were used as input data, and eye protrusion values ​​were used as label values.

[0374] In this case, the 5-fold cross validation method was used.

[0375] In this case, Pearson Correlation and MAE (mm) were as follows.

[0376] Pearson Correlation MAE (mm) Left eye 0.79 1.37 Right eye 0.79 1.38

[0377] Comparing the results of Experimental Example 4 with the results of Experimental Example 3, it was confirmed that estimating the eye protrusion value using an image that included the subject's nose bridge showed higher accuracy. In addition, comparing the results of Experimental Example 4 with the results of Experimental Examples 1 and 2, it was confirmed that estimating the eye protrusion value using an image that included both eyes of the subject rather than one eye and included the nose bridge rather than one that did not include the nose bridge showed higher accuracy.

[0378]

[0379] (5) Experimental example 5

[0380] An eye protrusion value estimation model was trained using an artificial neural network using 1,136 visible light images (see Fig. 14) showing both eyes of the subject, 1,136 depth images corresponding to each of the 1,136 visible light images, and the eye protrusion value of one eye corresponding to each of the 1,136 visible light images as a data set.

[0381] In this case, visible light images and depth images were used as input data, and eye protrusion values ​​were used as label values.

[0382] In this case, the 5-fold cross validation method was used.

[0383] In this case, Pearson Correlation and MAE (mm) were as follows.

[0384] Pearson Correlation MAE (mm) Left eye 0.78 1.42 Right eye 0.80 1.34

[0385] Comparing the results of Experimental Example 5 with the results of Experimental Examples 1 and 2, it was confirmed that estimating the eye protrusion value using an image that included the subject's nose bridge showed higher accuracy than an image that did not include the nose bridge.

[0386] (6) Experimental example 6

[0387] An eye protrusion value estimation model was trained using an artificial neural network using 1,136 visible light images (see Fig. 11) showing both eyes and the bridge of the nose of the subject, 1,136 depth images corresponding to each of the 1,136 visible light images, and the eye protrusion value of one eye corresponding to each of the 1,136 visible light images as a data set.

[0388] In this case, visible light images and depth images were used as input data, and the protrusion value of one eye was used as a single label value.

[0389] In this case, a method was used in which the learning data set and the test data set were randomly divided into a ratio of 7:3 and a single experiment was conducted.

[0390] In this case, Pearson Correlation, R 2 , and MAE (mm) were as follows.

[0391] Pearson CorrelationR 2 MAE(mm)Left eye0.790.621.40Right eye0.790.621.40

[0392] (7) Experimental Example 7

[0393] An eye protrusion value estimation model was trained using an artificial neural network using 1,136 visible light images (see Fig. 11) showing both eyes and the bridge of the nose of the subject, 1,136 depth images corresponding to each of the 1,136 visible light images, and the eye protrusion values ​​of each of the two eyes corresponding to each of the 1,136 visible light images as a data set.

[0394] In this case, visible light images and depth images were used as input data, and the protrusion values ​​of each eye were used as multi-label values.

[0395] In this case, a method was used in which the learning data set and the test data set were randomly divided into a ratio of 7:3 and a single experiment was conducted.

[0396] In this case, Pearson Correlation, R 2 , and MAE (mm) were as follows.

[0397] Pearson CorrelationR 2 MAE(mm)Left eye0.770.591.51Right eye0.770.591.45

[0398] (8) Experimental Example 8

[0399] An eye protrusion value estimation model was trained using an artificial neural network using 1,136 visible light images (see Fig. 11) showing both eyes and the bridge of the nose of the subject, 1,136 depth images corresponding to each of the 1,136 visible light images, and the eye protrusion value of one eye corresponding to each of the 1,136 visible light images as a data set.

[0400] In this case, visible light images and depth images were used as input data, and eye protrusion values ​​were used as label values.

[0401] In this case, the 5-fold cross validation method was used.

[0402] In this case, Pearson Correlation, R, is calculated for cases where the eye protrusion value is estimated by acquiring one visible light image and for cases where the eye protrusion value is estimated by acquiring three visible light images. 2 , MAE (mm) and ICC were as follows.

[0403] Pearson CorrelationR 2MAE (mm) ICC Image 1 sheet 0.77 0.59 1.24 0.71 Image 3 sheets 0.78 0.61 1.22 0.72

[0404] Through the results of Experimental Example 8, it was confirmed that acquiring two or more visible light images can estimate the eye protrusion value more accurately than acquiring one image.

Claims

1. Obtaining an image in which at least one eye of a subject appears, wherein the image includes a plurality of pixels to which values corresponding to at least one of brightness and color are assigned; Perform preprocessing on the above image to obtain a preprocessed image; Obtaining a depth image corresponding to the preprocessed image by using the preprocessed image in a pre-learned depth image generation model, wherein the depth image includes a plurality of pixels, and each of the plurality of pixels of the depth image is assigned a depth value representing a relative distance of an object corresponding to each of the pixels of the preprocessed image; Estimating the eye protrusion value for the eye of the subject by applying both the preprocessed image and the depth image to a pre-learned eye protrusion value estimation model; A method to estimate the value of eye protrusion.

2. In paragraph 1, The above ocular protrusion value estimation model is, A learning preprocessing image generated by preprocessing an image in which at least one eye of the first subject appears, a learning depth image corresponding to the learning preprocessing image, and learning using the eye protrusion value of the first subject. A method to estimate the value of eye protrusion.

3. In paragraph 1, The above image includes one eye of the subject, The above eye protrusion value is the eye protrusion value shown in the image. A method to estimate the value of eye protrusion.

4. In paragraph 1, The above image includes both eyes of the subject, The above eye protrusion value is a set of the protrusion value of the left eye of the subject and the protrusion value of the right eye of the subject shown in the image. A method to estimate the value of eye protrusion.

5. In paragraph 1, The above image includes both eyes of the subject, The above preprocessed image includes a first image corresponding to the left eye and a second image corresponding to the right eye among the two eyes of the subject, The eye protrusion value estimated by applying the first image above is the protrusion value of the right eye of the subject shown in the image, The eye protrusion value estimated by applying the second image above is the protrusion value of the left eye of the subject shown in the image. A method to estimate the value of eye protrusion.

6. In paragraph 1, The above image shows the entire face area including both eyes of the subject. Performing preprocessing on the above image to generate the above preprocessed image, generating the preprocessed image by cropping the image to at least a portion of the area where both eyes of the subject appear; A method to estimate the value of eye protrusion.

7. In paragraph 1, The above image shows the entire face area including at least one eye and the bridge of the nose of the subject, Performing preprocessing on the above image to generate the above preprocessed image, generating the preprocessed image by cropping the image to at least a portion of the area where the one eye and the bridge of the nose of the subject appear; A method to estimate the value of eye protrusion.

8. In paragraph 1, The above depth values are, A value between a preset minimum and maximum value, If the object represented in at least one pixel of the preprocessed image corresponding to the pixel of the depth image to which the depth value is assigned is the closest object appearing in the preprocessed image, then the maximum value is If the above object is the furthest object that appears in the above preprocessed image, then it is the minimum value, The closer the object is to the preprocessed image, the closer the value is to the maximum. The closer the object is to the minimum value, the farther away it is from the preprocessed image. A method to estimate the value of eye protrusion.

9. In paragraph 1, The above depth values are, A value between a preset minimum and maximum value, If the object represented in at least one pixel of the preprocessed image corresponding to the pixel of the depth image to which the depth value is assigned is the closest object appearing in the preprocessed image, then the minimum value is If the above object is the furthest object that appears in the above preprocessed image, then the maximum value is The closer the object is to the preprocessed image, the closer the value is to the minimum. The closer the object is to the maximum value, the farther away it is from the preprocessed image. A method to estimate the value of eye protrusion.

10. In paragraph 1, The above ocular protrusion value estimation model is, It has an artificial neural network structure, The values assigned to each pixel of the above preprocessed image are processed to obtain a first intermediate result, Processing the values assigned to each pixel of the depth image above to obtain a second intermediate result, By connecting the first intermediate result and the second intermediate result, a third intermediate result is obtained, Processing the third intermediate result and outputting the eye protrusion value, A method to estimate the value of eye protrusion.

11. In paragraph 1, The above ocular protrusion value estimation model is, The values assigned to each pixel of the above preprocessed image are processed by the first layer having the first ResNet structure to obtain the first intermediate result, The values assigned to each pixel of the depth image are processed by the second layer having the first ResNet structure to obtain a second intermediate result, By connecting the first intermediate result and the second intermediate result, a third intermediate result is obtained, Processing the third intermediate result with a third layer having a second ResNet structure to output the eye protrusion value. A method to estimate the value of eye protrusion.

12. In paragraph 1, The above image shows the entire face area of the subject, including both eyes of the subject, The image above is, i) The degree of smile on the face is within a predetermined level; ii) The face has a left-right rotation angle within a predetermined angle range; iii) The face has an up-down rotation angle within a predetermined angle range, and iv) The above face is located within a predetermined distance; Satisfying the condition that includes at least one of the following: A method to estimate the value of eye protrusion.

13. In paragraph 12, Obtaining the image captured by the above visible light camera, Providing a shooting guide for obtaining the visible light image satisfying the above conditions; further comprising; A method to estimate the value of eye protrusion.

14. In paragraph 1, further comprising providing the estimated eye protrusion value to the user device; A method to estimate the value of eye protrusion.

15. In paragraph 1, If the estimated eye protrusion value is greater than a preset threshold value, providing a visit guidance message for the subject; A method to estimate the value of eye protrusion.

16. In paragraph 15, The above threshold value is determined based on at least one of the race and facial shape of the subject. A method to estimate the value of eye protrusion.

17. In paragraph 1, Obtaining past eye protrusion values for the above object; and If the difference between the estimated eye protrusion value and the past eye protrusion value is greater than or equal to a preset threshold value, providing a visit guidance message for the subject; A method to estimate the value of eye protrusion.

18. In paragraph 1, Further comprising: providing information to determine at least one of thyroid ophthalmopathy severity and thyroid ophthalmopathy activity for the subject based on the estimated eye extrusion value; A method to estimate the value of eye protrusion.

19. In paragraph 1, Further comprising: providing information to determine whether at least one of drug treatment and surgery is necessary for the subject based on the estimated eye protrusion value; A method to estimate the value of eye protrusion.

20. In paragraph 1, Further comprising: providing information to determine the extent of surgery for the subject based on the estimated eye protrusion value; A method to estimate the value of eye protrusion.

21. In paragraph 1, Obtaining the above image is done by: Comprising obtaining at least a first image and a second image for the above object, The first image and the second image are obtained under the same shooting conditions. A method to estimate the value of eye protrusion.

22. In paragraph 21, Obtaining the above preprocessed image, obtaining the depth image, and estimating the eye protrusion value, It is performed once for the first image above, It is performed once for the above second image, Estimating the average value of the first eye protrusion value for the first image and the second eye protrusion value for the second image as the eye protrusion value on the day the image was acquired. A method to estimate the value of eye protrusion.

Images