Method for hospital visit guidance for medical treatment for active thyroid eye disease, and system for performing same

Abstract

To provide a computer-implemented method of predicting thyroid eye disease.SOLUTION: A method comprises: preparing a conjunctival hyperemia prediction model, a conjunctival edema prediction model, a lacrimal edema prediction model, an eyelid redness prediction model, and an eyelid edema prediction model; obtaining a facial image of an object; obtaining a first processed image and a second processed image from the facial image, wherein the first processed image is different from the second processed image; obtaining predicted values for each of a conjunctival hyperemia, a conjunctival edema and a lacrimal edema by applying the first processed image to the conjunctival hyperemia prediction model, the conjunctival edema prediction model, and the lacrimal edema prediction model; and obtaining predicted values for each of an eyelid redness and an eyelid edema by applying the second processed image to the eyelid redness prediction model and the eyelid edema prediction model.SELECTED DRAWING: Figure 28

Description

[Technical field]

[0001] The present disclosure relates to a method for clinic visit guidance for the medical treatment of active thyroid eye disease, and a system for carrying out the method. [Background technology]

[0002] Eye diseases are diseases that occur in the eye and the surrounding area. Many people around the world suffer from eye diseases, which can cause significant inconvenience in daily life, such as impairing vision in severe cases, and therefore it is necessary to monitor the occurrence or extent of eye diseases.

[0003] On the other hand, eye disease may be one of several complications caused by other diseases, for example, thyroid eye disease is a complication caused by thyroid dysfunction.

[0004] When thyroid eye disease worsens, the eyeballs protrude and cannot be treated without surgery. Therefore, early diagnosis of thyroid eye disease is very important for its treatment. However, since thyroid eye disease does not show obvious prognostic symptoms, it is difficult to diagnose the disease early. In the medical community, efforts have been made to diagnose thyroid eye disease early through an evaluation method using the Clinical Activity Score (CAS), which was proposed in 1989.

[0005] A total of seven items are considered in determining the clinical activity score for thyroid eye disease: 1) spontaneous retrobulbar pain, 2) pain on attempting to gaze upward or downward, 3) eyelid redness, 4) conjunctival redness, 5) eyelid swelling, 6) conjunctival swelling, and 7) swelling of the lacrimal caruncle.

[0006] In order to determine the clinical activity score, it is essential that the individual visits the hospital or clinic in person and that the doctor conducts an examination through interview and macroscopic observation. For example, spontaneous retrobulbar pain and pain when attempting to gaze upward or downward can be examined through interview by the doctor, and eyelid redness, conjunctival redness, eyelid swelling, conjunctival swelling, and lacrimal caruncle swelling can be examined through macroscopic observation by the doctor. The macroscopic examination and interview method by the doctor to determine the clinical activity score requires, as a prerequisite, that the patient visits the hospital in person to diagnose thyroid eye disease, which makes it difficult to diagnose thyroid eye disease early.

[0007] It is therefore desirable to develop methods that allow individuals to more easily and quickly recognize their risk of eye disease without an in-person clinic visit so that continuous monitoring can be performed, and to inform patients of their risk of eye disease so as to induce them to visit a clinic if necessary. Summary of the Invention [Problem to be solved by the invention]

[0008] The disclosure herein is directed to providing a learning model that can be used to predict clinical activity scores for thyroid eye disease by using images captured with digital cameras available to the general public rather than professional medical diagnostic devices.

[0009] Additionally, the disclosure herein is directed to providing methods and systems for enabling the general public to continuously monitor clinical activity scores for thyroid eye disease without the assistance of a physician and an in-person hospital visit.

[0010] The disclosure herein is further directed to providing a method for recommending a clinic visit for medical treatment for active thyroid eye disease according to the results of monitoring the clinical activity score, and a system for carrying out the method.

[0011] The technical problems to be solved by the present application are not limited to the above-mentioned technical problems, and other technical problems not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings. [Means for solving the problem]

[0012] According to one aspect of the present application, a computer-implemented method for predicting thyroid eye disease is disclosed. The method includes preparing a conjunctival hyperemia prediction model, a conjunctival edema prediction model, a caruncle edema prediction model, an eyelid redness prediction model, and an eyelid edema prediction model; acquiring a face image of a subject; acquiring a first processed image and a second processed image from the face image, where the first processed image is different from the second processed image; applying the first processed image to the conjunctival hyperemia prediction model, the caruncle edema prediction model, and the caruncle edema prediction model to obtain predicted values ​​for each of conjunctival hyperemia, chemosis, and caruncle edema; applying the second processed image to the eyelid redness prediction model and the eyelid edema prediction model to obtain predicted values ​​for each of eyelid redness and eyelid edema; and determining a likelihood that the subject has thyroid eye disease based on the predicted values ​​for conjunctival hyperemia, chemosis, caruncle edema, eyelid redness, and eyelid edema. Here, the first processed image is an image in which an area corresponding to the inside of the iris contour and an area corresponding to the outside of the eye contour are masked, and cropped along a first area including the contour of the eye based on position information of pixels corresponding to the contour of the iris included in the eye and position information of pixels corresponding to the contour of the eye, and here, the second processed image is an image cropped along a second area larger than the first area based on position information of pixels corresponding to the contour of the iris included in the eye and position information of pixels corresponding to the contour of the eye.

[0013] In some embodiments, the position information of the pixels corresponding to the contour of the iris included in the eye and the position information of the pixels corresponding to the contour of the eye are obtained by a segmentation model.

[0014] In some embodiments, the first processed image includes a first processed left eye image and a first processed right eye image, and the second processed image includes a second processed left eye image and a second processed right eye image.

[0015] In some embodiments, the conjunctival hyperemia prediction model comprises a left eye conjunctival hyperemia prediction model and a right eye conjunctival hyperemia prediction model, the chemokine prediction model comprises a left eye conjunctival chemokine prediction model and a right eye conjunctival chemokine prediction model, the caruncle edema prediction model comprises a left eye conjunctival edema prediction model and a right eye conjunctival edema prediction model, the eyelid redness prediction model comprises a left eyelid redness prediction model and a right eyelid redness prediction model, and the eyelid edema prediction model comprises a left eyelid edema prediction model and a right eyelid edema prediction model.

[0016] In some embodiments, the predicted value for conjunctival hyperemia is determined based on a result obtained by inputting the first processed left eye image into the left eye conjunctival hyperemia prediction model and a result obtained by inputting the first processed right eye image into the right eye conjunctival hyperemia prediction model, the predicted value for chemoconjunctival edema is determined based on a result obtained by inputting the first processed left eye image into the left eye conjunctival edema prediction model and a result obtained by inputting the first processed right eye image into the right eye conjunctival edema prediction model, and the predicted value for caruncular edema is determined based on a result obtained by inputting the first processed left eye image into the left eye conjunctival edema prediction model and a result obtained by inputting the first processed right eye image into the right eye conjunctival edema prediction model, and a result obtained by inputting the first processed right eye image into the right eye caruncle edema prediction model, the predicted value for eyelid redness is determined based on a result obtained by inputting the second processed left eye image into the left eyelid redness prediction model and a result obtained by inputting the second processed right eye image into the right eyelid redness prediction model, and the predicted value for eyelid edema is determined based on a result obtained by inputting the second processed left eye image into the left eyelid edema prediction model and a result obtained by inputting the second processed right eye image into the right eyelid edema prediction model.

[0017] In some embodiments, the method further comprises processing one of the first processed left eye image and the first processed right eye image by left-right flipping, and processing one of the second processed left eye image and the second processed right eye image by left-right flipping.

[0018] In some embodiments, the predicted value for the conjunctival hyperemia is determined based on results obtained by inputting the image that has been flipped left-right and by inputting the image that is not flipped left-right into the conjunctival hyperemia prediction model, the predicted value for the chemoconstriction ... caruncle edema is determined based on results obtained by inputting the image that has been flipped left-right and by inputting the image that is not flipped left-right into the caruncle edema prediction model, the predicted value for the eyelid redness is determined based on results obtained by inputting the image that has been flipped left-right and by inputting the image that is not flipped left-right into the eyelid hyperconstriction prediction model, and the predicted value for the eyelid edema is determined based on results obtained by inputting the image that has been flipped left-right and by inputting the image that is not flipped left-right into the eyelid edema prediction model.

[0019] In some embodiments, the method further comprises resizing the first processed left eye image and the first processed right eye image, and resizing the second processed left eye image and the second processed right eye image. Effect of the Invention

[0020] According to the disclosure herein, clinical activity scores for thyroid eye disease can be predicted using images acquired through digital cameras available to the general public, rather than professional medical diagnostic devices.

[0021] Furthermore, the disclosure herein allows the general public to continuously monitor their clinical activity scores for thyroid eye disease without the assistance of a physician and without an in-person hospital visit, with a hospital visit being recommended when necessary. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 illustrates a system for predicting a clinical activity score for thyroid eye disease according to one embodiment described herein.

[0023] [Diagram 2] FIG. 2 is a block diagram illustrating a user terminal provided in the present application.

[0024] [Diagram 3] FIG. 2 is a block diagram illustrating a server according to the present application.

[0025] [Figure 4] FIG. 2 shows the eye and surrounding tissues exposed to the outside so that they are captured by the camera when a picture of the face is taken using the camera.

[0026] [Diagram 5] FIG. 2 is a diagram showing an eyeball exposed to the outside.

[0027] [Figure 6] FIG. 1 is a diagram showing the contour of an eye.

[0028] [Figure 7] FIG. 2 shows the exposed cornea.

[0029] [Figure 8] FIG. 1 shows the exposed conjunctiva.

[0030] [Figure 9] FIG. 2 is a diagram showing a face image and a binocular image.

[0031] [Figure 10] FIG. 2 is a diagram showing a left eye image and a right eye image.

[0032] [Figure 11] FIG. 13 shows Xmax, Xmin, Ymax, and Ymin of contour pixels.

[0033] [Figure 12] FIG. 11 is a diagram showing a determined second cropping region.

[0034] [Figure 13] FIG. 11 is a diagram showing an example of a second cropped image.

[0035] [Figure 14] FIG. 11 is a diagram showing an example of a third cropped image.

[0036] [Figure 15] FIG. 1 illustrates iris segmentation. [Figure 16] FIG. 1 illustrates iris segmentation.

[0037] [Figure 17] FIG. 1 illustrates eye contour segmentation.

[0038] [Figure 18] FIG. 11 is a diagram showing an example of a first masking image.

[0039] [Figure 19] FIG. 11 is a diagram showing an example of a second masking image.

[0040] [Figure 20] 1A to 1C are diagrams showing various examples of an original image and a horizontally inverted image. [Figure 21] 1A to 1C are diagrams showing various examples of an original image and a horizontally inverted image. [Figure 22] 1A to 1C are diagrams showing various examples of an original image and a horizontally inverted image.

[0041] [Figure 23] 1 is a flowchart showing a method for predicting conjunctival hyperemia.

[0042] [Figure 24]1 is a flowchart showing a method for predicting chemosis.

[0043] [Diagram 25] 1 is a flow chart showing a method for predicting caruncular edema.

[0044] [Figure 26] 1 is a flowchart showing a method for predicting eyelid redness.

[0045] [Figure 27] 1 is a flowchart showing a method for predicting eyelid edema.

[0046] [Figure 28] FIG. 1 shows a method for predicting clinical activity score for thyroid eye disease.

[0047] [Figure 29] FIG. 1 illustrates a method for serially monitoring clinical activity scores for thyroid eye disease and for recommending clinic visits based on the former. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0048] The above-mentioned objects, features, and advantages of the present application will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. Furthermore, various modifications may be made to the present application, and various embodiments of the present application may be practiced. Accordingly, the following detailed description of specific embodiments will be given with reference to the accompanying drawings.

[0049] Throughout the specification, the same reference numerals refer to the same elements in principle. Furthermore, elements having the same functions within the same scope shown in the drawings of the embodiment are described using the same reference numerals, and redundant descriptions are omitted.

[0050] Detailed descriptions of well-known functions or configurations of the present application will be omitted if they are deemed to obscure the nature and spirit of the present application. Furthermore, throughout this specification, the terms first, second, etc. are used only to distinguish one element from another.

[0051] Furthermore, the terms "module" and "section" used to list elements in the following description are used solely for ease of writing the specification, and are not intended to have different special meanings or functions, and therefore may be used individually or interchangeably.

[0052] In the following embodiments, expressions used in the singular form also include expressions in the plural form unless it has a clearly different meaning in the context.

[0053] In the following embodiments, it should be understood that terms such as "comprise", "have", and the like are intended to indicate the presence of features or elements disclosed in the specification, and are not intended to exclude the possibility that one or more other features or elements may be added.

[0054] The size of elements in the drawings may be exaggerated or reduced for convenience of explanation. For example, any size and thickness of each element shown in the drawings is shown for convenience of explanation, and the present disclosure is not limited thereto.

[0055] In cases where particular embodiments are otherwise implemented, certain processes may be performed out of the order described. For example, two processes described as successive may be performed substantially simultaneously or may proceed in the reverse order from that described.

[0056] In the following embodiments, when elements are referred to as being connected to each other, the elements are directly connected to each other, or the elements are indirectly connected to each other with an intervening element between them. For example, in this specification, when elements are referred to as being electrically connected to each other, the elements are directly electrically connected to each other, or the elements are indirectly electrically connected to each other with an intervening element between them.

[0057]

[0058] According to one aspect of the present application, a computer-implemented method for predicting thyroid eye disease is disclosed. The method includes preparing a conjunctival hyperemia prediction model, a conjunctival edema prediction model, a caruncle edema prediction model, an eyelid redness prediction model, and an eyelid edema prediction model; acquiring a face image of a subject; acquiring a first processed image and a second processed image from the face image, where the first processed image is different from the second processed image; applying the first processed image to the conjunctival hyperemia prediction model, the caruncle edema prediction model, and the caruncle edema prediction model to obtain predicted values ​​for each of conjunctival hyperemia, chemosis, and caruncle edema; applying the second processed image to the eyelid redness prediction model and the eyelid edema prediction model to obtain predicted values ​​for each of eyelid redness and eyelid edema; and determining a likelihood that the subject has thyroid eye disease based on the predicted values ​​for conjunctival hyperemia, chemosis, caruncle edema, eyelid redness, and eyelid edema. Here, the first processed image is an image in which an area corresponding to the inside of the iris contour and an area corresponding to the outside of the eye contour are masked, and cropped along a first area including the contour of the eye based on position information of pixels corresponding to the contour of the iris included in the eye and position information of pixels corresponding to the contour of the eye, and here, the second processed image is an image cropped along a second area larger than the first area based on position information of pixels corresponding to the contour of the iris included in the eye and position information of pixels corresponding to the contour of the eye.

[0059] In some embodiments, the position information of the pixels corresponding to the contour of the iris included in the eye and the position information of the pixels corresponding to the contour of the eye are obtained by a segmentation model.

[0060] In some embodiments, the first processed image includes a first processed left eye image and a first processed right eye image, and the second processed image includes a second processed left eye image and a second processed right eye image.

[0061] In some embodiments, the conjunctival hyperemia prediction model comprises a left eye conjunctival hyperemia prediction model and a right eye conjunctival hyperemia prediction model, the chemokine prediction model comprises a left eye conjunctival chemokine prediction model and a right eye conjunctival chemokine prediction model, the caruncle edema prediction model comprises a left eye conjunctival edema prediction model and a right eye conjunctival edema prediction model, the eyelid redness prediction model comprises a left eyelid redness prediction model and a right eyelid redness prediction model, and the eyelid edema prediction model comprises a left eyelid edema prediction model and a right eyelid edema prediction model.

[0062] In some embodiments, the predicted value for conjunctival hyperemia is determined based on a result obtained by inputting the first processed left eye image into the left eye conjunctival hyperemia prediction model and a result obtained by inputting the first processed right eye image into the right eye conjunctival hyperemia prediction model, the predicted value for chemoconjunctival edema is determined based on a result obtained by inputting the first processed left eye image into the left eye conjunctival edema prediction model and a result obtained by inputting the first processed right eye image into the right eye conjunctival edema prediction model, and the predicted value for caruncular edema is determined based on a result obtained by inputting the first processed left eye image into the left eye conjunctival edema prediction model and a result obtained by inputting the first processed right eye image into the right eye conjunctival edema prediction model, and a result obtained by inputting the first processed right eye image into the right eye caruncle edema prediction model, the predicted value for eyelid redness is determined based on a result obtained by inputting the second processed left eye image into the left eyelid redness prediction model and a result obtained by inputting the second processed right eye image into the right eyelid redness prediction model, and the predicted value for eyelid edema is determined based on a result obtained by inputting the second processed left eye image into the left eyelid edema prediction model and a result obtained by inputting the second processed right eye image into the right eyelid edema prediction model.

[0063] In some embodiments, the method further comprises processing one of the first processed left eye image and the first processed right eye image by left-right flipping, and processing one of the second processed left eye image and the second processed right eye image by left-right flipping.

[0064] In some embodiments, the predicted value for the conjunctival hyperemia is determined based on results obtained by inputting the image that has been flipped left-right and by inputting the image that is not flipped left-right into the conjunctival hyperemia prediction model, the predicted value for the chemoconstriction ... caruncle edema is determined based on results obtained by inputting the image that has been flipped left-right and by inputting the image that is not flipped left-right into the caruncle edema prediction model, the predicted value for the eyelid redness is determined based on results obtained by inputting the image that has been flipped left-right and by inputting the image that is not flipped left-right into the eyelid hyperconstriction prediction model, and the predicted value for the eyelid edema is determined based on results obtained by inputting the image that has been flipped left-right and by inputting the image that is not flipped left-right into the eyelid edema prediction model.

[0065] In some embodiments, the method further comprises resizing the first processed left eye image and the first processed right eye image, and resizing the second processed left eye image and the second processed right eye image.

[0066]

[0067] According to the present application, a system for predicting a user's Clinical Activity Score (CAS) for thyroid eye disease and for providing the user with guidance on the need for a clinician visit based on the CAS is disclosed.

[0068]

[0069] 1. System-wide

[0070] (1) System hardware construction

[0071] FIG. 1 is a diagram illustrating a system for predicting a clinical activity score for thyroid eye disease according to one embodiment described herein.

[0072] Referring to FIG. 1, a system 1 includes a plurality of user terminals 10 and a server 20 .

[0073] The multiple user terminals 10 and the server 20 will now be described in detail.

[0074]

[0075] [(2) Functions of the user device]

[0076] A plurality of user terminals 10 transmit information to and receive information from a server 20 via various networks.

[0077] The multiple user terminals 10 acquire images of the user's upper eyelid, lower eyelid, and eyeball exposed by the upper and lower eyelids (hereinafter referred to as eye images). The multiple user terminals 10 may perform necessary processing on the acquired eye images, or may transmit the acquired eye images or the processed eye images to the server 20.

[0078] A number of user terminals 10 may receive from the server 20 the predicted results for the clinical activity scores processed by the server 20 .

[0079]

[0080] [(3) Server Functions]

[0081] The server 20 transmits information to and receives information from a plurality of user terminals 10 via various networks.

[0082] The server 20 may receive eye images from multiple user terminals 10. Here, the server 20 may process the eye images. Alternatively, the server 20 may receive processed eye images.

[0083] The server 20 may obtain a predicted outcome for the user's clinical activity score for thyroid eye disease based on the processed eye image.

[0084] The server 20 may transmit the prediction results for the clinical activity score to multiple user terminals 10.

[0085]

[0086] (4) System software construction

[0087] In order for System 1 to operate, several software components are required.

[0088] To implement communication between the user terminals 10 and the server 20, terminal software needs to be installed on the user terminals 10, and server software needs to be installed on the server 20.

[0089] A variety of pre-processing algorithms may be used to perform the necessary pre-processing of the eye images.

[0090] A number of learning models may be used to predict clinical activity scores based on pre-processed ocular images.

[0091] The pre-processing algorithms may be executed by terminal software installed on the user terminal 10 or by software installed on the server 20. Alternatively, some of the pre-processing algorithms may be executed by the user terminal 10 and others by the server 20.

[0092] The multiple learning models may be executed by software installed on the server 20. Alternatively, the multiple learning models may be executed by terminal software installed on the user terminal 10. Alternatively, some of the multiple learning models may be executed by the user terminal 10, and others may be executed by the server 20.

[0093] (5) User device elements

[0094] FIG. 2 is a block diagram illustrating a user terminal as described herein.

[0095] Referring to FIG. 2, the user terminal 10 described in the present application includes an output unit 110, a communication unit 120, a memory 130, a camera 140, and a controller 150.

[0096] The output unit 110 outputs various types of information according to control commands of the controller 150. According to one embodiment, the output unit 110 may include a display 112 for visually outputting information to the user. Alternatively, although not shown in the drawings, the output unit 110 may include a speaker for audibly outputting information to the user and a vibration motor for tactilely outputting information to the user.

[0097] The communication unit 120 may include a wireless communication module and / or a wired communication module, where examples of the wireless communication module include a Wi-Fi (registered trademark) communication module, a cellular communication module, and the like.

[0098] The memory 130 stores executable codes readable by the controller 150, processed result values, necessary data, etc. Examples of the memory 130 may include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, etc. The memory 130 may store the above-mentioned terminal software, and may store executable codes for implementing the above-mentioned various pre-processing algorithms and / or learning models. Furthermore, the memory 130 may store eye images acquired through the camera 140, pre-processed eye images, etc.

[0099] The camera 140 is a digital camera and may include an image sensor and an image processor. The image sensor is a device for converting an optical image into an electrical signal, and may be provided as a chip incorporating a number of photodiodes. Examples of the image sensor may include a charge-coupled device (CCD), a complementary metal-oxide semiconductor (CMOS), and the like. Meanwhile, the image processor may perform image processing on the captured result and generate image information.

[0100] The controller 150 may include at least one processor, where each of the processors may perform a certain operation by executing at least one instruction stored in the memory 130. Specifically, the controller 150 may process information according to terminal software, pre-processing algorithms, and / or learning models running on the user terminal 10. Meanwhile, the controller 150 controls the overall operation of the user terminal 10.

[0101] Although not shown in the drawings, the user terminal 10 may have a user input unit. The user terminal 10 may receive various types of information required for the operation of the user terminal 10 from a user through the user input unit.

[0102]

[0103] (6) Server elements

[0104] FIG. 3 is a block diagram illustrating a server according to the present application.

[0105] Referring to FIG. 3, the server 20 described herein includes a communication unit 210, a memory 220, and a controller 230.

[0106] The communication unit 210 may include a wireless communication module and / or a wired communication module, where examples of the wireless communication module include a Wi-Fi communication module, a cellular communication module, and the like.

[0107] The memory 220 stores executable code readable by the controller 230, processed result values, necessary data, etc. Examples of the memory 220 may include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, etc. The memory 220 may store the server software described above, and may store executable code for implementing the various pre-processing algorithms and / or learning models described above. Furthermore, the memory 220 may store eye images received from the user terminal 10, pre-processed eye images, etc.

[0108] The controller 230 may include at least one processor, where each of the processors may perform a certain operation by executing at least one instruction stored in the memory 220. In particular, the controller 230 may process information according to server software, pre-processing algorithms, and / or learning models running on the server 20. Meanwhile, the controller 230 controls the overall operation of the server 20.

[0109]

[0110] In order to make the technology described herein clearer and easier to understand, the following provides a brief description of the eye, eyeball, and tissues surrounding the eyeball, including the upper eyelid, lower eyelid, and caruncle, and defines terms related to the eye and surrounding area used in this specification.

[0111]

[0112] 2. Structure of the eye and definition of terms

[0113] [(1) Eyeball and surrounding tissues]

[0114] FIG. 4 illustrates the eye and surrounding tissues exposed to the outside so that they are captured by the camera when a picture of the face is taken using the camera.

[0115] FIG. 4 shows the eyelids (upper and lower), the lower eyelid, the caruncle, and the conjunctiva and cornea partially exposed and partially covered by the upper and lower eyelids, and the caruncle.

[0116] In general, the eye or eyeball is larger than shown in Figure 4. However, the eyeball is protected from the outside by tissues such as the upper and lower eyelids, and therefore, even when a person has their eye open, only a portion of the eyeball is exposed to the outside.

[0117]

[0118] (2) Definitions of terms

[0119] Conjunctiva, white of the eye

[0120] Hereinafter, the conjunctiva corresponds generally to the location of the white of the eye, and so the terms conjunctiva and white of the eye may be used interchangeably.

[0121] cornea, iris

[0122] Hereinafter, the cornea corresponds generally to the location of the iris of the eye, and so the terms cornea and iris may be used interchangeably, whereas in this specification the term "iris" is used to include the pupil area.

[0123] eyelids

[0124] The eyelids are two folds of skin, upper and lower, that cover the front of the eyeball. Eyelids are also called palpebrae. The upper eyelid is called the upper eyelid, and the lower eyelid is called the lower eyelid. The outer and inner surfaces of the skin are the conjunctiva, and between them is the tarsal plate, which contains the muscles that move the eyelids and the sebaceous glands, the meibomian glands, thus maintaining the shape of the eyelids. The eyelids protect the eyeball and at the same time clean it with tears when you blink, or make the cornea shiny and clear.

[0125] eyebrow

[0126] Eyebrows are the arched mass of hair that grows along the bone ridge above the eye.

[0127] eyelashes

[0128] Eyelashes are hairs approximately 10 mm long that are found on the margins of the upper and lower eyelids.

[0129] Exposed eyeball

[0130] Hereinafter, the term "exposed eyeball" refers to the part of the eye that is not covered by the upper eyelid, the lower eyelid, and the caruncle when a person has his / her eye open, i.e., the part exposed to the outside by the upper eyelid, the lower eyelid, and the caruncle. For example, the inside of the dotted line shown in FIG. 5 is called the "exposed eyeball."

[0131] Eye contour

[0132] Hereinafter, the term "eye contour" refers to the contour of a portion including both the eyeball and the caruncle area exposed to the outside when a person has his / her eye open. That is, the contour of the area that is a combination of the eyeball and the caruncle exposed to the outside is called the "eye contour." For example, the dotted line shown in FIG. 6 is called the "eye contour."

[0133] Exposed cornea (exposed iris)

[0134] Hereinafter, the term "exposed cornea" refers to a portion of the cornea that is not covered by the upper and lower eyelids when a person has his / her eyes open, i.e., a portion of the cornea that is exposed to the outside by the upper and lower eyelids. For example, the inside of the dotted line shown in FIG. 7 is called the "exposed cornea."

[0135] Exposed conjunctiva (exposed white of the eye)

[0136] Hereinafter, the term "exposed conjunctiva" refers to the part of the conjunctiva that is not covered by the upper eyelid, the lower eyelid, and the caruncle, i.e., the part of the conjunctiva that is exposed to the outside by the upper eyelid, the lower eyelid, and the caruncle when a person has his / her eyes open. For example, the area inside the dotted line shown in FIG. 8 is called the "exposed conjunctiva."

[0137]

[0138] In the following, various image pre-processing algorithms are described for performing the image pre-processing described in this application.

[0139]

[0140] 3. Image Preprocessing Algorithms

[0141] [(1) The necessity of image preprocessing]

[0142] The present application is directed to providing a learning model for predicting clinical activity scores for thyroid eye disease by using images acquired by digital cameras available to the general public rather than professional medical diagnostic devices.

[0143] For this reason, in predicting the clinical activity score for thyroid eye disease, images of the eye and tissues near the eye that can be easily obtained by ordinary people should be used. For example, the image analysis uses digital images obtained by a digital camera or a camera built into a smartphone that can be easily used by ordinary people, rather than digital images obtained by a special medical device used in a medical institution.

[0144] In this environment, it is difficult to standardize the digital images acquired by users, and various types of pre-processing of the acquired images are required to more accurately and quickly recognize the digital images acquired by users.

[0145]

[0146] [(2) First cropping (binocular image cropping)]

[0147] Images used in predicting the clinical activity score for thyroid eye disease should include the left eye, the right eye, and the surrounding area.

[0148] However, for faster and more accurate analysis, it is more efficient to use an image of only the eyes and surrounding areas (hereinafter referred to as a binocular image) that does not contain multiple unnecessary areas (e.g., areas corresponding to the nose, mouth, forehead, etc.) in the image analysis, rather than an image of the entire face (hereinafter referred to as a facial image).

[0149] Therefore, it is necessary to extract an image including both eyes (left eye / right eye) (hereinafter referred to as a binocular image) from an image of the entire face acquired by the user (hereinafter referred to as a face image).

[0150] For example, a binocular image (an inner region of a rectangle marked with a dotted line) as shown in Fig. 9(b) may be obtained from a face image acquired by a user shown in Fig. 9(a). Hereinafter, obtaining a binocular image from a face image acquired by a user in this manner is referred to as binocular image cropping or first cropping.

[0151]

[0152] (3) The need to apply additional cropping methods

[0153] The inventors of the present application have constructed a system using the first cropped image (binocular image) described above to predict scores on five items related to thyroid eye disease through a predictive model described below, but found that the accuracy of the predictions was low.

[0154] The inventors of the present application determined that the binocular image contained many regions unnecessary for analysis, leading to low prediction accuracy, and that it was necessary to obtain a more precise cropped image, i.e., it was determined that it would be more efficient to obtain and use separate left-eye and right-eye images as shown in (b) of Fig. 10 than to use a binocular image as shown in (a) of Fig. 10, in which the left and right eyes are included in one image.

[0155]

[0156] (4) The need to apply different cropping methods

[0157] It was explained that five of the seven items used to assess the clinical activity score for thyroid eye disease are assessed by a physician through a macroscopic observation of the user's eye and surrounding area. The five items are:

[0158] 1) Conjunctival congestion (redness of the conjunctiva),

[0159] 2) Chemotherapy (swelling of the conjunctiva),

[0160] 3) Caruncular edema (swelling of the lacrimal caruncle),

[0161] 4) Redness of the eyelids, and

[0162] 5) Eyelid edema (swelling of the eyelids).

[0163] As described below, the independent prediction model provided herein has been applied to five symptoms to assess clinical activity scores for thyroid eye disease.

[0164] Although there may be methods that use images with different cropping methods applied through five independent prediction models, the inventors of the present application have determined that sufficient prediction accuracy can be obtained by applying an image cropping method for analyzing the conjunctiva and caruncle and an image cropping method for analyzing the eyelids.

[0165]

[0166] [(5) Second cropping (cropping based on eye contour)]

[0167] In the following, cropping based on eye contour (second cropping) is described. The second cropping may be applied to both right-eye and left-eye images, but for convenience, a description is given based on obtaining a right-eye cropped image.

[0168] Second cropping objective

[0169] The second cropping is to generate images that will be used as input images for models for predicting whether there is conjunctival redness, whether there is conjunctival swelling, and whether there is caruncular swelling, among other predictive models described below. The second cropping is to generate images in which information about the cornea and caruncle is maximized and information about other regions is minimized.

[0170] Input image

[0171] The second cropping may be applied to the face image or the binocular image (the first cropped image).

[0172] Eye contour detection

[0173] According to one embodiment, pixels corresponding to the boundary between the upper eyelid and the eyeball and the boundary between the lower eyelid and the eyeball may be detected to detect the contour of the right eye. Furthermore, pixels corresponding to the intersection of the upper eyelid and the lower eyelid may be detected to detect the contour of the right eye. Furthermore, pixels corresponding to the caruncle may be detected to detect the contour of the right eye.

[0174] According to another embodiment, the contour pixels corresponding to the outermost portion of the eye contour may be detected using an eye contour segmentation model, which will be described later.

[0175] Determine the maximum and minimum X and Y coordinates of the contour pixels

[0176] When judging the detected pixel, the maximum X coordinate value X max , the minimum X coordinate value X min , the maximum Y coordinate value max , and the minimum Y coordinate value Y min is determined.

[0177] Figure 11 shows the X of the contour pixels. max , X min , Y max , and Y min FIG.

[0178] Cropping area determination

[0179] The determined X of the contour pixel max , X min , Y max , and Y min Based on this, a rectangle having the following four points as vertices is generated, and the area contained inside the rectangle is determined as the cropping area.

[0180] (X min ,Y max ),

[0181] (X max ,Y max ),

[0182] (X max ,Y min ), and

[0183] (X min ,Y min )

[0184] FIG. 12 is a diagram showing the determined second cropping region.

[0185] As explained above, a second cropping region may be determined in the same manner for the left eye.

[0186] Generate a second cropped image

[0187] A second cropping area is determined, and based on the determined second cropping area, a second cropped image (a second right eye cropped image and a second left eye cropped image) may be generated from the face image or binocular image by using the pixels contained inside the second cropping area determined as described above, as shown in FIG. 13.

[0188]

[0189] Hereinafter, the terms "second right-eye cropped image" and "right-eye contour cropped image" may be used interchangeably, and the terms "second left-eye cropped image" and "left-eye contour cropped image" may be used interchangeably.

[0190] Furthermore, without specific mention below, the term "second cropped image (or contour cropped image)" may mean either the second right-eye cropped image or the second left-eye cropped image, or may mean both, depending on the context.

[0191] The second cropped image means an image cropped with respect to the "eye contour". If the cropped image is generated such that the top, bottom, rightmost and leftmost pixels of the "eye contour" are included in the cropping area, the cropped image generated in a manner different from the above-mentioned manner is called the second cropped image (contour cropped image).

[0192] On the other hand, the X and Y coordinate values ​​in this application have different sizes and directions depending on the relative positions to the reference point, so the terms maximum and minimum should be understood in a relative sense, not an absolute sense. That is, when the position of the origin of the coordinate system changes, the maximum value of the above-mentioned X coordinate value may be the minimum value of the X coordinate value in the coordinate system whose origin has changed, and the minimum value of the X coordinate value may be the maximum value of the X coordinate value in the coordinate system whose origin has changed. This can be equally applied to the Y coordinate value.

[0193]

[0194] [(6) Third cropping (eyelid-inclusive cropping)]

[0195] Below, eyelid-inclusive cropping (third cropping) is described. The third cropping may be applied to both right-eye and left-eye images, but for convenience, a description is given based on obtaining a right-eye cropped image.

[0196] Third cropping objective

[0197] The third cropping is to generate an image that will be used as an input image for a model for predicting whether there is eyelid redness and a model for predicting whether there is eyelid swelling, among other predictive models described later. The third cropping is to include information about the eyelids in the image. Here, instead of cropping only the pixels corresponding to the eyelids, it may be better to generate a cropped image to include all pixels included in the eye contour. This is because inferences and judgments are required regarding color values ​​to predict whether there is eyelid redness, and color values ​​of pixels corresponding to the iris and / or white of the eye may be used.

[0198] Input image

[0199] The third cropping may be applied to the face image or the binocular image (the first cropped image).

[0200] Detecting the eye contour and determining the maximum and minimum X and Y coordinates of the contour pixels

[0201] According to one embodiment, the eye contour detection method described in the second cropping may be applied as is, or the contour pixel corresponding to the outermost part of the eye contour may be detected. When determining the detected pixel, the maximum X coordinate value X max , the minimum X coordinate value X min , the maximum Y coordinate value max , and the minimum Y coordinate value Y min may be determined.

[0202] Cropping area determination #1

[0203] Determined Y max Value and Y min A first extension value Y determined according to a predetermined criterion based on the value e Y max A second extension value Y that is added to the value and determined according to a predetermined criterion e ' is Y minThe third cropping region may be determined in a manner similar to that described above for determining the second cropping region.

[0204] That is, a rectangle having the following four vertices is generated, and the area contained inside the rectangle is determined as the third cropping area.

[0205] (X min ,Y max +Y e )

[0206] (X max ,Y max +Y e ),

[0207] (X max ,Y min -Y e '), and

[0208] (X min ,Y min -Y e ')

[0209] When the third cropping region is determined in this manner, more pixels corresponding to the upper eyelid and the lower eyelid may be included in the image compared to the second cropping region.

[0210] Here, the first expansion value and the second expansion value may be, but are not necessarily, the same.

[0211] On the other hand, the criterion for determining the first extension value and the criterion for determining the second extension value may be the same, but are not necessarily the same.

[0212] The first and second expansion values ​​may be determined based on the size of the second cropping region. For example, the first and second expansion values ​​may be determined using a number of pixels corresponding to a length calculated by multiplying a horizontal length of the second cropping region by an expansion percentage. As another example, the first and second expansion values ​​may be determined using a number of pixels corresponding to a length calculated by multiplying a vertical length of the second cropping region by an expansion percentage.

[0213] Here, the specific percentage may be any one of the following: 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, and 60%.

[0214] The expansion percentage used in determining the first expansion value and the expansion percentage used in determining the second expansion value may, but are not necessarily the same, and may be different from one another.

[0215] If the horizontal length of the second cropping region is used to determine the first extension value, the horizontal length may also be used in determining the second extension value, but is not limited to this, and the vertical length may be used in determining the second extension value.

[0216] Cropping area determination #2

[0217] Determined X max Value and X min Based on the value, the first width expansion value X we is determined according to a predetermined criterion and a second width expansion value X we is determined.

[0218] Determined Y max Value and Y minBased on the value, the first height expansion value Y is calculated according to a predetermined criterion. he is determined according to a predetermined criterion, and a second height expansion value Y he is determined.

[0219] First width expansion value X we X max The value obtained by adding to the value, X min Value to second width extension value X we ', the value obtained by subtracting the first height extension value Y he Y max The value obtained by adding to the value, and Y min Value to second height extension value Y he Based on the value obtained by subtracting , the third cropping region may be determined in a manner similar to the above-described method for determining the second cropping region.

[0220] That is, a rectangle having the following four vertices is generated, and the area contained inside the rectangle is determined as the third cropping area.

[0221] (X min -X we ',Y max +Y he ),

[0222] (X max +X we ,Y max +Y he ),

[0223] (X max +X we ,Y min -Y he '), and

[0224] (X min -X we ',Y min -Y he ')

[0225] When the third cropping region is determined in this manner, more pixels corresponding to the upper eyelid and the lower eyelid may be included in the image compared to the second cropping region.

[0226] Moreover, the cropped image contains more pixels in the left-right direction than the image cropped by the method "Cropping Area Determination #1". As a result, the cropped image contains more information about the upper and lower eyelids. Because the width of the upper and lower eyelids is usually wider than the width of the eyeball exposed to the outside, more pixels corresponding to the upper and lower eyelids are included through vertical and horizontal expansion.

[0227] Here, the first width expansion value and the second width expansion value may be, but are not necessarily, the same.

[0228] On the other hand, the criterion for determining the first height expansion value and the criterion for determining the second height expansion value may be the same, but are not necessarily the same.

[0229] The first and second width expansion values ​​may be determined based on the size of the second cropping region. For example, the first and second width expansion values ​​may be determined using a number of pixels corresponding to a length calculated by multiplying a horizontal length of the second cropping region by an expansion percentage. As another example, the first and second width expansion values ​​may be determined using a number of pixels corresponding to a length calculated by multiplying a vertical length of the second cropping region by an expansion percentage.

[0230] Here, the specific percentage may be any one of the following: 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, and 60%.

[0231] The expansion percentage used in determining the first width expansion value and the expansion percentage used in determining the second width expansion value may, but are not necessarily the same, and may be different from one another.

[0232] When the horizontal length of the second cropping region is used to determine the first width expansion value, the horizontal length may also be used in determining the second expansion value, but is not limited to this, and the vertical length may be used in determining the second width expansion value.

[0233] Meanwhile, the method for determining the first height expansion value and the second height expansion value is the same as the above-mentioned method for determining the first expansion value and the second expansion value, and therefore detailed description is omitted.

[0234] Generate the third cropped image

[0235] A third cropping area is determined, and based on the determined third cropping area, a third cropped image (a third right eye cropped image and a third left eye cropped image) may be generated from the face image or the binocular image by using pixels contained inside the third cropping area determined as described above, as shown in FIG. 14.

[0236] For reference, FIG. 14(a) shows a third cropped image cropped by the above-mentioned method “Cropping Area Determination #1”, and FIG. 14(b) shows a third cropped image cropped by the above-mentioned method “Cropping Area Determination #2”.

[0237]

[0238] Hereinafter, the terms "third right-eye cropped image" and "right-eyelid-inclusive cropped image" may be used interchangeably, and the terms "third left-eye cropped image" and "left-eyelid-inclusive cropped image" may be used interchangeably.

[0239] Furthermore, without specific mention below, the term "third cropped image (or eyelid-inclusive cropped image)" may mean either a third right-eye cropped image or a third left-eye cropped image, or may mean both, depending on the context.

[0240] The third cropped image means an image generated such that the image includes information about the eyelids. A cropped image generated in a manner different from that described above is called a third cropped image (eyelid-inclusive cropped image) if the border of the cropping region is determined such that the pixels corresponding to the eyelids are additionally included.

[0241]

[0242] [(7) Iris Segmentation]

[0243] In the following, iris segmentation is described.

[0244] Iris segmentation may be performed by a model that distinguishes regions corresponding to the iris or cornea from the eye and peripheral images.

[0245] By using iris segmentation, pixels corresponding to the iris in the image may be inferred, as shown in FIG.

[0246] By using iris segmentation, pixels corresponding to the externally exposed iris in the image may be inferred, as shown in FIG.

[0247] A model for iris segmentation may receive a face image as input data and may output a "1" for pixels in the face image that are inferred to correspond to an iris and a "0" for other pixels.

[0248] The iris segmentation model may be trained using training data including face images and images in which pixels corresponding to the iris in the face images have a pixel value of "1" and the remaining pixels have a pixel value of "0".

[0249] Although face images have been described as being received as input data and subjected to iris segmentation, the iris segmentation may also be performed using the binocular images discussed above as input data.

[0250]

[0251] [(8) Eye contour segmentation]

[0252] In the following, eye contour segmentation is described.

[0253] Eye contour segmentation may be performed by a model that distinguishes a region corresponding to the inside of the eye contour from the eye and peripheral images.

[0254] By using eye contour segmentation, pixels corresponding to the inside of the eye contour in the image may be inferred, as shown in FIG.

[0255] A model for eye contour segmentation may receive a face image as input data and may output a "1" for pixels inferred to be pixels corresponding to inside the eye contour in the face image and a "0" for other pixels.

[0256] The eye contour segmentation model may be trained using training data including facial images and images in which pixels corresponding to inside the eye contours in the facial images have pixel values ​​of "1" and the remaining pixels have pixel values ​​of "0".

[0257]

[0258] [(9) Masking]

[0259] First masking

[0260] In the present application, first masking means that the information reflected in pixel values ​​corresponding to areas of the image excluding pixels corresponding to the conjunctiva and the caruncle is removed.

[0261] Removing information reflected in pixel values ​​means changing the pixel values ​​of the pixels from which the information is to be removed to a specific predefined value, for example, all pixel values ​​of the pixels from which the information is to be removed may be changed to 0.

[0262] The first masking may be performed before the images are input into a model for predicting symptoms associated with the conjunctiva and caruncle, among other predictive models for predicting clinical activity scores for thyroid eye disease.

[0263] Second Masking

[0264] In the present application, the second masking means that the information reflected in the pixel values ​​corresponding to the areas in the image that correspond to the exposed cornea (exposed iris) is removed.

[0265] The second masking may be performed before the images are input into a model for predicting symptoms related to the eyelids (upper eyelid and lower eyelid), among the above-mentioned predictive models for predicting clinical activity scores for thyroid eye disease.

[0266] First masking method

[0267] The first masking may be performed on a first masking target image, which is one selected from the group of a face image, a first cropped image (binocular image), and a second cropped image (contour cropped image).

[0268] A first masking image may be generated based on the first masking target image, the eye contour segmentation result, and the iris segmentation result, for example, pixel values ​​excluding pixels corresponding to the inside of the eye contour and pixel values ​​corresponding to the iris (or the externally exposed iris) may be removed from the first masking target image.

[0269] FIG. 18 is a diagram showing an example of the first masking image.

[0270] Second masking method

[0271] The second masking may be performed on a second masking target image, which is one selected from the group of the face image, the first cropped image (binocular image), and the third cropped image (eyelid-inclusive cropped image).

[0272] A second masking image may be generated based on the second masking target image, the eye contour segmentation result, and the iris segmentation result, for example, pixel values ​​of pixels corresponding to the inside of the eye contour and at the same time corresponding to the iris (or the externally exposed iris) may be removed from the second masking target image.

[0273] FIG. 19 is a diagram showing an example of the second masking image.

[0274] Another embodiment of the first masking

[0275] According to the above description, the first masking removes all pixel values ​​of pixels corresponding to areas excluding pixels corresponding to the conjunctiva and caruncle, but the first masking does not have to remove pixel values ​​of pixels corresponding to the cornea (iris) if necessary.

[0276] However, since iris color can vary with race, removing the pixel values ​​of pixels that correspond to the iris is advantageous for faster learning and higher accuracy.

[0277] Second Masking Option

[0278] According to the above description, the second masking is described as removing all pixel values ​​of pixels corresponding to the iris, but it is also acceptable that the second masking is not performed at all.

[0279] However, since the color of the iris may vary with race, it is advantageous for faster learning and higher accuracy to remove the pixel values ​​of the pixels corresponding to the iris by performing a second masking.

[0280]

[0281] [(10) Flip Horizontal]

[0282] The necessity of left-right inversion

[0283] According to the method of predicting clinical activity score for thyroid eye disease provided herein, instead of using binocular images, cropped images of the left and right eyes are used.

[0284] On the other hand, the eye contour is asymmetric: for example, for the right eye, the caruncle is at the left edge of the right eye, but the natural meeting point of the upper and lower eyelids is at the right edge of the right eye.

[0285] Therefore, it is more effective to use the learning models trained for the right eye and the learning models trained for the left eye separately for faster learning and more accurate prediction.

[0286] However, when the left eye is rotated to become the right eye based on the axis of symmetry between the left and right eyes, the shape features of the right and left eyes are similar to each other.

[0287] Therefore, according to the present application, either the right eye or the left eye is used without mirroring, and the other eye is used mirroring, so that only one learning model can be used.

[0288] Flip horizontally

[0289] Flipping an image horizontally (transforming the left and right sides of an image) means that if a first pixel value corresponds to pixel (a+△,Y) in an image and a second pixel value corresponds to pixel (a-△,Y), then the pixel value of (a+△,Y) is changed from a first pixel value to a second pixel value and the pixel value of (a-△,Y) is changed from the second pixel value to the first pixel value, according to a horizontal reference line (X=a) that runs through the image to be flipped and divides the image in half horizontally.

[0290] Horizontally inverted target image

[0291] It is sufficient to mirror-flip either the left-eye image or the right-eye image, which one is mirror-flipped being determined according to which one of the left-eye image and the right-eye image is based on when the predictive model, described below, is trained.

[0292] Alternatively, left-right flipping may be performed on an image that has had both masking and cropping (second cropping or third cropping) performed, or left-right flipping may be performed on an image that has only had cropping performed but no masking performed.

[0293] 20 to 22 are diagrams showing various examples of original images and left-right inverted images.

[0294] Flip Horizontal Option

[0295] However, as explained above, left-right flipping is applied to combine the prediction model for the left eye and the prediction model for the right eye, and therefore, if the prediction model for the left eye and the prediction model for the right eye are realized as different models, the left-right flip pre-processing may be omitted.

[0296]

[0297] [(11) Resize]

[0298] Need for resizing

[0299] As explained above, if the image is cropped about the eye contour and the cropped image is used, the size of the eye varies from person to person and the cropped image will vary in size from person to person.

[0300] On the other hand, if the left-eye and right-eye images are cropped and acquired independently, the left-eye cropped image and the right-eye cropped image of the same person will differ from each other due to the size difference between the left and right eyes.

[0301] For this reason, before the eye images are input into the prediction models described below, they need to be resized to a standard size corresponding to the respective prediction model.

[0302] Standard size for each predictive model

[0303] The standard sizes corresponding to the first to fifth prediction models may be different from each other.

[0304] The standard size corresponding to the predictive model using the second cropped image as the input image may be the same.

[0305] The standard size corresponding to the predictive model using the third cropped image as the input image may be the same.

[0306] The standard size corresponding to a predictive model that uses the second cropped image as an input image may be different from the standard size corresponding to a predictive model that uses the third cropped image as an input image.

[0307] Alternatively, the standard sizes corresponding to the first to fifth prediction models may be the same.

[0308] How to resize

[0309] The size of the resize target image is adjusted to fit the standard size.

[0310] If the width or height of the resized target image is larger than the width or height of the standard size, the width or height of the resized target image may be decreased.

[0311] If the width or height of the resized target image is smaller than the width or height of the standard size, the width or height of the resized target image may be increased.

[0312] In resizing, the aspect ratio of the image before resizing may be different from the aspect ratio of the image after resizing.

[0313]

[0314] 4. Predictive Model

[0315] [(1) First prediction model]

[0316] Purpose and operation of the first predictive model

[0317] The first prediction model is a model for predicting whether or not conjunctival hyperemia is present.

[0318] The first predictive model may receive an eye image as input data and may output a probability value that the conjunctiva captured in the input eye image is hyperemic.

[0319] When the first prediction model includes a first left eye prediction model and a first right eye prediction model, the first left eye prediction model may receive a left eye image and output a probability value that the conjunctiva captured in the left eye image is hyperemic, and the first right eye prediction model may receive a right eye image and output a probability value that the conjunctiva captured in the right eye image is hyperemic.

[0320] If the first prediction model is not dualized and is realized as a single model, the first prediction model may receive either a right eye image or a left eye image and output a probability value that the conjunctiva captured in the input image is conjunctival hyperemia, and may receive the other image and output a probability value that the conjunctiva captured in the input image is conjunctival hyperemia.

[0321] The eye image may be an image that has been preprocessed according to the preprocessing algorithms described above.

[0322] For example, the eye image may be an image that has been pre-processed according to a second cropping.

[0323] As another example, the eye image may be an image that has undergone pre-processing, including a second cropping and resizing.

[0324] As yet another example, the eye image may be an image that has undergone pre-processing including second cropping, first masking, and resizing.

[0325] As yet another example, the eye image may be an image that has undergone pre-processing including second cropping, first masking, left-right flipping, and resizing.

[0326] In this specification, the first prediction model may be referred to as a conjunctival hyperemia prediction model.

[0327] Training the first predictive model

[0328] To train the first prediction model, a plurality of training data sets may be prepared. The training data sets may include eye images and evaluation values ​​for conjunctival hyperemia captured in the eye images. The eye images may be images preprocessed by the above-mentioned preprocessing algorithm. For example, the eye images may be images preprocessed including a second cropping, a first masking, and a resizing.

[0329] An artificial intelligence model may be prepared to train the first predictive model.

[0330] Examples of artificial intelligence models may be Support Vector Machine (SVM), Random Forest, Gradient Boosting algorithm, ResNet, VGG, GoogLeNet, MobileNet, and Vision Transformer.

[0331] Next, eye images contained in the prepared multiple training data sets are input into the artificial intelligence model, and training is performed using the evaluation values ​​corresponding to each of the input eye images and the output values ​​output from the artificial intelligence model.

[0332] When the first prediction model includes a first left eye prediction model and a first right eye prediction model, the multiple training data sets for training the first left eye prediction model may include a left eye image and an evaluation value related to conjunctival hyperemia captured in the left eye image, and the multiple training data sets for training the first right eye prediction model may include a right eye image and an evaluation value related to conjunctival hyperemia captured in the right eye image. On the other hand, in order to increase the number of training data sets, the multiple training data sets for training the first left eye prediction model may include a right eye image subjected to left-right flipping processing and an evaluation value related to conjunctival hyperemia captured in the right eye image, and the multiple training data sets for training the first right eye prediction model may include a left eye image subjected to left-right flipping processing and an evaluation value related to conjunctival hyperemia captured in the left eye image.

[0333] In the case where the first prediction model is not dualized, but rather intended to be realized as one model, the multiple training data sets may include a right eye image and an evaluation value regarding conjunctival hyperemia captured in the right eye image, or may include a left eye image on which left-right flipping has been performed and an evaluation value regarding conjunctival hyperemia captured in the left eye image. Alternatively, the multiple training data sets may include a left eye image and an evaluation value regarding conjunctival hyperemia captured in the left eye image, or may include a right eye image on which left-right flipping has been performed and an evaluation value regarding conjunctival hyperemia captured in the right eye image.

[0334] On the other hand, in training the first prediction model, in order to predict whether or not there is conjunctival congestion without distinguishing between right eye images and left eye images, all right eye images, right eye images with left-right inversion, left eye images, and left eye images with left-right inversion are used as training data for training one model.

[0335] For example, when the first prediction model includes a first left eye prediction model and a first right eye prediction model, the multiple training datasets for training the first left eye prediction model may include a left eye image and an evaluation value for conjunctival hyperemia captured in the left eye image; and a right eye image on which left-right flip has been performed and an evaluation value for conjunctival hyperemia captured in the right eye image, and the multiple training datasets for training the first right eye prediction model may include a right eye image and an evaluation value for conjunctival hyperemia captured in the right eye image; and a left eye image on which left-right flip has been performed and an evaluation value for conjunctival hyperemia captured in the left eye image.

[0336] If the first prediction model is not dualized, but rather is intended to be realized as a single model, the multiple training datasets may include a right eye image and an evaluation value for conjunctival hyperemia captured in the right eye image; a right eye image on which left-right inversion has been performed and an evaluation value for conjunctival hyperemia captured in the right eye image; and a left eye image and an evaluation value for conjunctival hyperemia captured in the left eye image; and a left eye image on which left-right inversion has been performed and an evaluation value for conjunctival hyperemia captured in the left eye image.

[0337]

[0338] [(2) Second prediction model]

[0339] Purpose and operation of the second predictive model

[0340] The second prediction model is a model for predicting whether or not chemosis is present.

[0341] The second predictive model may receive an eye image as input data and may output a probability value of the presence of chemosis captured in the input eye image.

[0342] When the second prediction model includes a second left eye prediction model and a second right eye prediction model, the second left eye prediction model may receive a left eye image and output a probability value of the presence of chemosis captured in the left eye image, and the second right eye prediction model may receive a right eye image and output a probability value of the presence of chemosis captured in the right eye image.

[0343] If the second predictive model is not dualized and is implemented as a single model, the second predictive model may receive either a right eye image or a left eye image and output a probability value of the presence of chemosis captured in the input image, and may receive the other image and output a probability value of the presence of chemosis captured in the input image.

[0344] The eye image may be an image that has been preprocessed according to the preprocessing algorithms described above.

[0345] For example, the eye image may be an image that has been pre-processed according to a second cropping.

[0346] As another example, the eye image may be an image that has undergone pre-processing, including a second cropping and resizing.

[0347] As yet another example, the eye image may be an image that has undergone pre-processing including second cropping, first masking, and resizing.

[0348] As yet another example, the eye image may be an image that has undergone pre-processing including second cropping, first masking, left-right flipping, and resizing.

[0349] In this specification, the second predictive model may be referred to as a chemosis predictive model.

[0350] Training a second predictive model

[0351] To train the second prediction model, a plurality of training data sets may be prepared. The training data sets may include eye images and evaluation values ​​for the presence of chemosis captured in the eye images. The eye images may be images preprocessed by the above-mentioned preprocessing algorithm. For example, the eye images may be images on which preprocessing including second cropping, first masking, and resizing has been performed.

[0352] An artificial intelligence model may be prepared to train the second predictive model.

[0353] Examples of artificial intelligence models may be Support Vector Machine (SVM), Random Forest, Gradient Boosting algorithm, ResNet, VGG, GoogLeNet, MobileNet, and Vision Transformer.

[0354] Next, eye images contained in the prepared multiple training data sets are input into the artificial intelligence model, and training is performed using the evaluation values ​​corresponding to each of the input eye images and the output values ​​output from the artificial intelligence model.

[0355] When the second prediction model includes a second left eye prediction model and a second right eye prediction model, the multiple training data sets for training the second left eye prediction model may include a left eye image and an evaluation value regarding the presence of chemophore captured in the left eye image, and the multiple training data sets for training the second right eye prediction model may include a right eye image and an evaluation value regarding the presence of chemophore captured in the right eye image. On the other hand, in order to increase the number of training data sets, the multiple training data sets for training the second left eye prediction model may include a right eye image subjected to left-right flipping processing and an evaluation value regarding the presence of chemophore captured in the right eye image, and the multiple training data sets for training the second right eye prediction model may include a left eye image subjected to left-right flipping processing and an evaluation value regarding the presence of chemophore captured in the left eye image.

[0356] In the case where the second prediction model is not dualized but is intended to be realized as one model, the multiple training data sets may include a right eye image and an evaluation value regarding the presence of chemosis captured in the right eye image, or may include a left eye image on which left-right flipping has been performed and an evaluation value regarding the presence of chemosis captured in the left eye image. Alternatively, the multiple training data sets may include a left eye image and an evaluation value regarding the presence of chemosis captured in the left eye image, or may include a right eye image on which left-right flipping has been performed and an evaluation value regarding the presence of chemosis captured in the right eye image.

[0357] On the other hand, in training the second prediction model, in order to predict whether or not there is chemosis without distinguishing between right eye images and left eye images, all right eye images, right eye images with left-right inversion, left eye images, and left eye images with left-right inversion are used as training data for training one model.

[0358] For example, when the second prediction model includes a second left eye prediction model and a second right eye prediction model, the multiple training datasets for training the second left eye prediction model may include a left eye image and an evaluation value for chemosis captured in the left eye image; and a right eye image on which left-right flipping has been performed and an evaluation value for chemosis captured in the right eye image, and the multiple training datasets for training the second right eye prediction model may include a right eye image and an evaluation value for chemosis captured in the right eye image; and a left eye image on which left-right flipping has been performed and an evaluation value for chemosis captured in the left eye image.

[0359] If the second predictive model is not dualized, but rather is intended to be realized as a single model, the multiple training datasets may include a right eye image and an evaluation value for chemosis captured in the right eye image; a right eye image on which left-right flipping has been performed and an evaluation value for chemosis captured in the right eye image; and a left eye image and an evaluation value for chemosis captured in the left eye image; and a left eye image on which left-right flipping has been performed and an evaluation value for chemosis captured in the left eye image.

[0360]

[0361] [(3) The third prediction model]

[0362] Purpose and operation of the third prediction model

[0363] The third prediction model is a model for predicting whether or not there is caruncular edema.

[0364] The third predictive model may receive an eye image as input data and may output a probability value of the presence of caruncular edema captured in the input eye image.

[0365] When the third prediction model includes a third left eye prediction model and a third right eye prediction model, the third left eye prediction model may receive a left eye image and output a probability value of the presence of caruncular edema captured in the left eye image, and the third right eye prediction model may receive a right eye image and output a probability value of the presence of caruncular edema captured in the right eye image.

[0366] If the third prediction model is not dualized and is realized as a single model, the third prediction model may receive either a right eye image or a left eye image and output a probability value of the presence of caruncular edema captured in the input image, and may receive the other image and output a probability value of the presence of caruncular edema captured in the input image.

[0367] The eye image may be an image that has been preprocessed according to the preprocessing algorithms described above.

[0368] For example, the eye image may be an image that has been pre-processed according to a second cropping.

[0369] As another example, the eye image may be an image that has undergone pre-processing, including a second cropping and resizing.

[0370] As yet another example, the eye image may be an image that has undergone pre-processing including second cropping, first masking, and resizing.

[0371] As yet another example, the eye image may be an image that has undergone pre-processing including second cropping, first masking, left-right flipping, and resizing.

[0372] Herein, the third predictive model may be referred to as a caruncular edema predictive model.

[0373] Training the third predictive model

[0374] To train the third prediction model, a plurality of training data sets may be prepared. The training data sets may include eye images and assessment values ​​for the presence of caruncular edema captured in the eye images. The eye images may be images preprocessed by the above-mentioned preprocessing algorithm. For example, the eye images may be images on which preprocessing including second cropping, first masking, and resizing has been performed.

[0375] An artificial intelligence model may be prepared for training the third predictive model.

[0376] Examples of artificial intelligence models may be Support Vector Machine (SVM), Random Forest, Gradient Boosting algorithm, ResNet, VGG, GoogLeNet, MobileNet, and Vision Transformer.

[0377] Next, eye images contained in the prepared multiple training data sets are input into the artificial intelligence model, and training is performed using the evaluation values ​​corresponding to each of the input eye images and the output values ​​output from the artificial intelligence model.

[0378] When the third prediction model includes a third left eye prediction model and a third right eye prediction model, the multiple training datasets for training the third left eye prediction model may include a left eye image and an evaluation value regarding the presence of caruncular edema captured in the left eye image, and the multiple training datasets for training the third right eye prediction model may include a right eye image and an evaluation value regarding the presence of caruncular edema captured in the right eye image. On the other hand, in order to increase the number of training datasets, the multiple training datasets for training the third left eye prediction model may include a right eye image subjected to left-right flipping processing and an evaluation value regarding the presence of caruncular edema captured in the right eye image, and the multiple training datasets for training the third right eye prediction model may include a left eye image subjected to left-right flipping processing and an evaluation value regarding the presence of caruncular edema captured in the left eye image.

[0379] In the case where the third prediction model is not dualized but rather intended to be realized as one model, the multiple training data sets may include a right eye image and an evaluation value regarding the presence of caruncular edema captured in the right eye image, or may include a left eye image on which left-right flipping has been performed and an evaluation value regarding the presence of caruncular edema captured in the left eye image. Alternatively, the multiple training data sets may include a left eye image and an evaluation value regarding the presence of caruncular edema captured in the left eye image, or may include a right eye image on which left-right flipping has been performed and an evaluation value regarding the presence of caruncular edema captured in the right eye image.

[0380] On the other hand, in training the third prediction model, in order to predict whether or not there is caruncular edema without distinguishing between right eye images and left eye images, all right eye images, right eye images with left-right inversion performed, left eye images, and left eye images with left-right inversion performed are used as training data for training one model.

[0381] For example, when the third prediction model includes a third left eye prediction model and a third right eye prediction model, the multiple training datasets for training the third left eye prediction model may include a left eye image and an evaluation value for caruncular edema captured in the left eye image; and a right eye image on which left-right flipping has been performed and an evaluation value for caruncular edema captured in the right eye image, and the multiple training datasets for training the third right eye prediction model may include a right eye image and an evaluation value for caruncular edema captured in the right eye image; and a left eye image on which left-right flipping has been performed and an evaluation value for caruncular edema captured in the left eye image.

[0382] If the third prediction model is not dualized, but rather is intended to be realized as a single model, the multiple training datasets may include a right eye image and an evaluation value for caruncular edema captured in the right eye image; a right eye image on which left-right flipping has been performed and an evaluation value for caruncular edema captured in the right eye image; and a left eye image and an evaluation value for caruncular edema captured in the left eye image; and a left eye image on which left-right flipping has been performed and an evaluation value for caruncular edema captured in the left eye image.

[0383]

[0384] [(4) Fourth prediction model]

[0385] Purpose and operation of the fourth prediction model

[0386] The fourth prediction model is a model for predicting whether or not eyelid redness will occur.

[0387] The fourth predictive model may receive an eye image as input data and may output a probability value of the presence of eyelid redness captured in the input eye image.

[0388] When the fourth prediction model includes a fourth left eye prediction model and a fourth right eye prediction model, the fourth left eye prediction model may receive a left eye image and output a probability value of the presence of eyelid redness captured in the left eye image, and the fourth right eye prediction model may receive a right eye image and output a probability value of the presence of eyelid redness captured in the right eye image.

[0389] If the fourth prediction model is not dualized and is implemented as a single model, the fourth prediction model may receive either a right eye image or a left eye image and output a probability value of the presence of eyelid redness captured in the input image, and may receive the other image and output a probability value of the presence of eyelid redness captured in the input image.

[0390] The eye image may be an image that has been preprocessed according to the preprocessing algorithms described above.

[0391] For example, the eye image may be an image that has been pre-processed according to a third cropping.

[0392] As another example, the eye image may be an image that has undergone pre-processing including a third cropping and resizing.

[0393] As yet another example, the eye image may be an image that has undergone pre-processing including third cropping, horizontal flipping, and resizing.

[0394] As yet another example, the eye image may be an image that has undergone pre-processing including a third cropping, a second masking, and a resizing.

[0395] As yet another example, the eye image may be an image that has undergone pre-processing including third cropping, second masking, left-right flipping, and resizing.

[0396] In this specification, the fourth prediction model may be referred to as an eyelid redness prediction model.

[0397] Training the fourth predictive model

[0398] To train the fourth prediction model, a plurality of training data sets may be prepared. The training data sets may include eye images and evaluation values ​​for eyelid redness captured in the eye images. The eye images may be images preprocessed by the above-mentioned preprocessing algorithm. For example, the eye images may be images preprocessed including a second cropping, a first masking, and a resizing.

[0399] An artificial intelligence model may be prepared to train the fourth predictive model.

[0400] Examples of artificial intelligence models may be Support Vector Machine (SVM), Random Forest, Gradient Boosting algorithm, ResNet, VGG, GoogLeNet, MobileNet, and Vision Transformer.

[0401] Next, eye images contained in the prepared multiple training data sets are input into the artificial intelligence model, and training is performed using the evaluation values ​​corresponding to each of the input eye images and the output values ​​output from the artificial intelligence model.

[0402] When the fourth prediction model includes a fourth left eye prediction model and a fourth right eye prediction model, the multiple training data sets for training the fourth left eye prediction model may include a left eye image and an evaluation value for eyelid redness captured in the left eye image, and the multiple training data sets for training the fourth right eye prediction model may include a right eye image and an evaluation value for eyelid redness captured in the right eye image. On the other hand, in order to increase the number of training data sets, the multiple training data sets for training the fourth left eye prediction model may include a right eye image subjected to left-right flipping processing and an evaluation value for eyelid redness captured in the right eye image, and the multiple training data sets for training the fourth right eye prediction model may include a left eye image subjected to left-right flipping processing and an evaluation value for eyelid redness captured in the left eye image.

[0403] In the case where the fourth prediction model is not dualized, but rather intended to be realized as one model, the multiple training data sets may include a right eye image and an evaluation value for eyelid redness captured in the right eye image, or may include a left eye image on which left-right flipping has been performed and an evaluation value for eyelid redness captured in the left eye image. Alternatively, the multiple training data sets may include a left eye image and an evaluation value for eyelid redness captured in the left eye image, or may include a right eye image on which left-right flipping has been performed and an evaluation value for eyelid redness captured in the right eye image.

[0404] On the other hand, in training the fourth prediction model, in order to predict whether or not there is eyelid redness without distinguishing between right eye images and left eye images, all right eye images, right eye images with left-right inversion, left eye images, and left eye images with left-right inversion are used as training data for training one model.

[0405] For example, when the fourth prediction model includes a fourth left eye prediction model and a fourth right eye prediction model, the multiple training datasets for training the fourth left eye prediction model may include a left eye image and an evaluation value for eyelid redness captured in the left eye image; and a right eye image on which left-right flipping has been performed and an evaluation value for eyelid redness captured in the right eye image, and the multiple training datasets for training the fourth right eye prediction model may include a right eye image and an evaluation value for eyelid redness captured in the right eye image; and a left eye image on which left-right flipping has been performed and an evaluation value for eyelid redness captured in the left eye image.

[0406] If the fourth predictive model is not dualized, but rather is intended to be realized as a single model, the multiple training data sets may include a right eye image and an evaluation value for eyelid redness captured in the right eye image; a right eye image subjected to left-right flipping and an evaluation value for eyelid redness captured in the right eye image; and a left eye image and an evaluation value for eyelid redness captured in the left eye image; and a left eye image subjected to left-right flipping and an evaluation value for eyelid redness captured in the left eye image.

[0407] [(5) Fifth prediction model]

[0408] Purpose and operation of the fifth prediction model

[0409] The fifth prediction model is a model for predicting whether or not eyelid edema is present.

[0410] The fifth predictive model may receive an eye image as input data and may output a probability value of the presence of eyelid edema captured in the input eye image.

[0411] When the fifth prediction model includes a fifth left eye prediction model and a fifth right eye prediction model, the fifth left eye prediction model may receive a left eye image and output a probability value of the presence of eyelid edema captured in the left eye image, and the fifth right eye prediction model may receive a right eye image and output a probability value of the presence of eyelid edema captured in the right eye image.

[0412] If the fifth predictive model is not dualized and is realized as a single model, the fifth predictive model may receive either a right eye image or a left eye image and output a probability value of the presence of eyelid edema captured in the input image, and may receive the other image and output a probability value of the presence of eyelid edema captured in the input image.

[0413] The eye image may be an image that has been preprocessed according to the preprocessing algorithms described above.

[0414] For example, the eye image may be an image that has been pre-processed according to a third cropping.

[0415] As another example, the eye image may be an image that has undergone pre-processing including a third cropping and resizing.

[0416] As yet another example, the eye image may be an image that has undergone pre-processing including third cropping, horizontal flipping, and resizing.

[0417] As yet another example, the eye image may be an image that has undergone pre-processing including a third cropping, a second masking, and a resizing.

[0418] As yet another example, the eye image may be an image that has undergone pre-processing including third cropping, second masking, left-right flipping, and resizing.

[0419] In this specification, the fifth predictive model may be referred to as an eyelid edema predictive model.

[0420] Training the fifth predictive model

[0421] To train the fifth prediction model, a plurality of training data sets may be prepared. The training data sets may include eye images and assessment values ​​for the presence of eyelid edema captured in the eye images. The eye images may be images preprocessed by the above-mentioned preprocessing algorithm. For example, the eye images may be images preprocessed including a third cropping, a second masking, and a resizing.

[0422] An artificial intelligence model may be prepared to train the fifth predictive model.

[0423] Examples of artificial intelligence models may be Support Vector Machine (SVM), Random Forest, Gradient Boosting algorithm, ResNet, VGG, GoogLeNet, MobileNet, and Vision Transformer.

[0424] Next, eye images contained in the prepared multiple training data sets are input into the artificial intelligence model, and training is performed using the evaluation values ​​corresponding to each of the input eye images and the output values ​​output from the artificial intelligence model.

[0425] When the fifth prediction model includes a fifth left eye prediction model and a fifth right eye prediction model, the plurality of training data sets for training the fifth left eye prediction model may include a left eye image and an evaluation value regarding the presence of eyelid edema captured in the left eye image, and the plurality of training data sets for training the fifth right eye prediction model may include a right eye image and an evaluation value regarding the presence of eyelid edema captured in the right eye image. On the other hand, in order to increase the number of training data sets, the plurality of training data sets for training the fifth left eye prediction model may include a right eye image subjected to left-right flipping processing and an evaluation value regarding the presence of eyelid edema captured in the right eye image, and the plurality of training data sets for training the fifth right eye prediction model may include a left eye image subjected to left-right flipping processing and an evaluation value regarding the presence of eyelid edema captured in the left eye image.

[0426] In the case where the fifth prediction model is not dualized, but rather intended to be realized as one model, the multiple training data sets may include a right eye image and an evaluation value regarding the presence of eyelid edema captured in the right eye image, or may include a left eye image on which left-right flipping has been performed and an evaluation value regarding the presence of eyelid edema captured in the left eye image. Alternatively, the multiple training data sets may include a left eye image and an evaluation value regarding the presence of eyelid edema captured in the left eye image, or may include a right eye image on which left-right flipping has been performed and an evaluation value regarding the presence of eyelid edema captured in the right eye image.

[0427] On the other hand, in training the fifth prediction model, in order to predict whether or not there is eyelid edema without distinguishing between right eye images and left eye images, all right eye images, right eye images with left-right inversion, left eye images, and left eye images with left-right inversion are used as training data for training one model.

[0428] For example, when the fifth prediction model includes a fifth left eye prediction model and a fifth right eye prediction model, the multiple training datasets for training the fifth left eye prediction model may include a left eye image and an evaluation value for eyelid edema captured in the left eye image; and a right eye image on which left-right flipping has been performed and an evaluation value for eyelid edema captured in the right eye image, and the multiple training datasets for training the fifth right eye prediction model may include a right eye image and an evaluation value for eyelid edema captured in the right eye image; and a left eye image on which left-right flipping has been performed and an evaluation value for eyelid edema captured in the left eye image.

[0429] If the fifth predictive model is not dualized, but rather is intended to be realized as a single model, the multiple training datasets may include a right eye image and an evaluation value for eyelid edema captured in the right eye image; a right eye image subjected to left-right flipping and an evaluation value for eyelid edema captured in the right eye image; and a left eye image and an evaluation value for eyelid edema captured in the left eye image; and a left eye image subjected to left-right flipping and an evaluation value for eyelid edema captured in the left eye image.

[0430]

[0431] The training of the predictive model may be performed by the electronic device, in particular by the server 20 described above. Moreover, the training of the predictive model by the electronic device or the server 20 refers to a series of processes that allow the output value of the predictive model for the input data to be similar to the output value labeled in the input data. To this end, the electronic device or the server 20 may use the difference between the output value of the predictive model and the labeled value to change the weight value of each of the nodes included in the predictive model. Here, the electronic device or the server 20 may determine the amount of change in the weight value of each of the nodes by using various feedback functions.

[0432]

[0433] The following methods are described below through the above-mentioned system 1: a method for predicting each symptom associated with a clinical activity score for thyroid eye disease by preprocessing an eye image and inputting the preprocessed eye image into the above-mentioned prediction model; a method for predicting a clinical activity score based on the prediction results of each symptom; and a method for monitoring the prediction results of the clinical activity score and providing guidance or recommendation to a user to visit a hospital for a medical examination according to the monitored results.

[0434]

[0435] 5. Method for predicting conjunctival congestion

[0436] The conjunctival hyperemia prediction method described herein may be performed by the server 20.

[0437] FIG. 23 is a flowchart showing a method for predicting conjunctival congestion.

[0438] Referring to FIG. 23, the server 20 acquires a facial image in step S100, preprocesses the acquired facial image in step S110, inputs the preprocessed image into the above-mentioned first prediction model (conjunctival hyperemia prediction model) in step S120, and acquires an output value of the first prediction model in step S130.

[0439] Acquiring face images

[0440] The server 20 acquires a face image in step S100. The server 20 may acquire the face image from the user terminal 10.

[0441] Facial image preprocessing

[0442] The server 20 may pre-process the facial image acquired in step S110. The server 20 may perform the iris segmentation, eye contour segmentation, masking, cropping, and resizing described above on the acquired facial image.

[0443] Segmentation Process

[0444] The server 20 may perform iris segmentation and eye contour segmentation, and the server 20 may therefore determine the pixels in the captured face image that correspond to the iris and the pixels that correspond to the inside of the eye contour. The server 20 may determine the coordinate values ​​of the pixels that correspond to the iris and the pixels that correspond to the inside of the eye contour.

[0445] Masking Process

[0446] The server 20 may perform a first masking on the face image based on the determined information about the pixels. Through the first masking process, the server 20 may remove pixel values ​​of pixels included in the face image, excluding pixels corresponding to the conjunctiva and caruncle exposed to the outside. Thus, pixel values ​​of pixels corresponding to the conjunctiva and caruncle of the left eye and the conjunctiva and caruncle of the right eye may remain as original pixel values, but pixel values ​​of pixels corresponding to the iris (or cornea) of the left eye, the iris (or cornea) of the right eye, outside the outline of the left eye, and outside the outline of the right eye may be removed or changed to other values.

[0447] Cropping

[0448] The server 20 may crop the masked face image. The server 20 may crop the masked face image to generate a left eye cropped image and a right eye cropped image. In executing the conjunctival hyperemia prediction method, the server 20 may use the second cropping (eye contour cropping) method among the cropping methods described above. The second cropping method has already been described in detail, so a detailed description thereof will be omitted here.

[0449] Resize and flip processing

[0450] The server 20 may resize the size of the left-eye cropped image and the size of the right-eye cropped image to a predetermined size.

[0451] On the other hand, if the first prediction model is not binarized but rather realized as one model, the server 20 may horizontally flip either the left-eye cropped image or the right-eye cropped image as described above. The server 20 does not horizontally flip the other of the left-eye cropped image and the right-eye cropped image. Here, it has been determined that the criteria for determining whether one of the left-eye image and the right-eye image is to be horizontally flipped is the same as the criteria applied when the first prediction model is trained. That is, in training the first prediction model, if the left-eye image is flipped and the right-eye image is not flipped, the server 20 will flip the left-eye image and will not flip the right-eye image as well.

[0452] As described above, in implementing the first prediction model, if the first prediction model is binarized into a first left eye prediction model and a first right eye prediction model, the server 20 does not need to perform left-right inversion processing.

[0453] On the other hand, while it has been described that when preprocessing is performed, segmentation, masking, cropping, resizing, and left-right flipping are performed, the order of these types of preprocessing may be changed within the scope capable of achieving the objective of the conjunctival hyperemia prediction method disclosed in the present application.

[0454] Input preprocessed images

[0455] The server 20 may input the preprocessed image into a first prediction model in step S120.

[0456] If the first prediction model is not dualized and is realized as a single model, the server 20 sequentially inputs the right eye preprocessed image and the left eye preprocessed image that is flipped left and right into the first prediction model.

[0457] In realizing the first prediction model, when the first prediction model is binarized into a first left eye prediction model and a first right eye prediction model, the server 20 inputs the left eye preprocessed image into the first left eye prediction model, and the right eye preprocessed image into the first right eye prediction model. Alternatively, the server 20 may input the left eye preprocessed image into the first left eye prediction model, may input the left eye preprocessed image that is left-right inverted into the first right eye prediction model, may input the right eye preprocessed image into the first right eye prediction model, and may input the right eye preprocessed image that is left-right inverted into the first left eye prediction model.

[0458] In implementing the first prediction model, if the first prediction model is not binarized but implemented as one model, and is simultaneously trained to be able to determine whether or not there is conjunctival hyperemia without distinguishing between the left eye image and the right eye image, the server 20 may input the left eye preprocessed image and the right eye preprocessed image to the first prediction model without flipping them horizontally. Alternatively, the server 20 may input the left eye preprocessed image, the left eye preprocessed image that is flipped horizontally, the right eye preprocessed image, and the right eye preprocessed image that is flipped horizontally to the first prediction model.

[0459] Conjunctival congestion prediction result

[0460] The server 20 may output a result value output from the first prediction model in step S130. The result value may be a predicted probability value for conjunctival hyperemia captured in the image. Based on a predefined threshold, if the predicted probability value is equal to or higher than the threshold, the server 20 determines that the conjunctiva is hyperemic, or if the predicted probability value is less than the threshold, the server 20 determines that the conjunctiva is not hyperemic.

[0461] The server 20 may obtain both the predicted results for the left eye and the predicted results for the right eye.

[0462] When the server 20 inputs a left-eye preprocessed image into a first left-eye prediction model, inputs a left-right-flipped left-eye preprocessed image into a first right-eye prediction model, inputs a right-eye preprocessed image into a first right-eye prediction model, and inputs a left-right-flipped right-eye preprocessed image into a first left-eye prediction model, the server 20 may obtain a prediction result for the left eye that takes into account both the result obtained by inputting the left-eye preprocessed image into the first left-eye prediction model and the result obtained by inputting the left-right-flipped left-eye preprocessed image into the first right-eye prediction model. Here, the server 20 may obtain a prediction result for the right eye that takes into account both the result obtained by inputting the right-eye preprocessed image into the first right-eye prediction model and the result obtained by inputting the right-left-flipped right-eye preprocessed image into the first left-eye prediction model.

[0463] For example, the server 20 may obtain a predicted result for the left eye based on whether an average value of a result obtained by inputting a left eye preprocessed image into a first left eye prediction model and a result obtained by inputting a left-right inverted left eye preprocessed image into a first right eye prediction model is equal to or higher than a threshold value.

[0464] As another example, if either the value of the result obtained by inputting a left eye preprocessed image into a first left eye prediction model or the value of the result obtained by inputting a left-right inverted left eye preprocessed image into a first right eye prediction model is equal to or higher than the above-mentioned threshold value, the server 20 may predict that the conjunctiva of the left eye is hyperemic.

[0465] As another example, if both the result obtained by inputting the left eye preprocessed image into the first left eye prediction model and the result obtained by inputting the left-right inverted left eye preprocessed image into the first right eye prediction model are equal to or higher than the above-mentioned threshold, the server 20 may predict that the conjunctiva of the left eye is hyperemic.

[0466] When the server 20 inputs a left eye preprocessed image, a left eye preprocessed image that is horizontally inverted, a right eye preprocessed image, and a right eye preprocessed image that is horizontally inverted into a first prediction model that is not binarized, the server 20 may obtain a prediction result for the left eye that takes into account both a result obtained by inputting the left eye preprocessed image into the first prediction model and a result obtained by inputting the left eye preprocessed image that is horizontally inverted into the first prediction model. Here, the server 20 may obtain a prediction result for the right eye that takes into account both a result obtained by inputting the right eye preprocessed image into the first prediction model and a result obtained by inputting the right eye preprocessed image that is horizontally inverted into the first prediction model.

[0467] For example, the server 20 may obtain a predicted result for the left eye based on whether an average value of a result obtained by inputting a left eye preprocessed image into a first prediction model and a result obtained by inputting a left-right inverted left eye preprocessed image into the first prediction model is equal to or higher than a threshold value.

[0468] As another example, if either the value of the result obtained by inputting a left eye pre-processed image into the first prediction model or the value of the result obtained by inputting a left eye pre-processed image that has been mirrored into the first prediction model is equal to or higher than the above-mentioned threshold value, the server 20 may predict that the conjunctiva of the left eye is hyperemic.

[0469] As another example, if both the results obtained by inputting a left eye pre-processed image into the first prediction model and the results obtained by inputting a left eye pre-processed image that has been flipped left side into the first prediction model are equal to or higher than the above-mentioned threshold value, the server 20 may predict that the conjunctiva of the left eye is hyperemic.

[0470] The above-described method may be similarly applied to determining whether or not there is conjunctival injection in the right eye.

[0471]

[0472] 6. Method for predicting conjunctival edema

[0473] The chemosis prediction method described herein may be performed by the server 20.

[0474] FIG. 24 is a flowchart showing a method for predicting chemosis.

[0475] Referring to FIG. 24, the server 20 acquires a facial image in step S200, preprocesses the acquired facial image in step S210, inputs the preprocessed image into the above-mentioned second prediction model (conjunctival edema prediction model) in step S220, and acquires an output value of the second prediction model in step S230.

[0476] This chemosis prediction method is the same as or very similar to the conjunctival congestion prediction method, except that a second prediction model is used instead of the first prediction model, and the final result value obtained is a prediction value regarding whether or not there is chemosis, and a detailed description of this chemosis prediction method will be omitted.

[0477]

[0478] 7. Method for predicting caruncular edema

[0479] The caruncular edema prediction method described herein may be performed by the server 20.

[0480] FIG. 25 is a flow chart showing a method for predicting caruncular edema.

[0481] Referring to FIG. 25, the server 20 acquires a facial image in step S300, preprocesses the acquired facial image in step S310, inputs the preprocessed image into the above-mentioned third prediction model (caruncular edema prediction model) in step S320, and acquires an output value of the third prediction model in step S330.

[0482] This caruncular edema prediction method is the same as or very similar to the conjunctival congestion prediction method, except that the third prediction model is used instead of the first prediction model, and a detailed description of this caruncular edema prediction method will be omitted.

[0483]

[0484] As described above, the conjunctival hyperemia prediction method, the conjunctival edema prediction method, and the caruncular edema prediction method use the same image pre-processing method, but differ from each other only in the prediction models into which the pre-processed images are input. Thus, after the image pre-processing described above, the images may be input into different prediction models.

[0485] However, while the caruncular edema prediction method has been described as using the same image pre-processing method as the conjunctival hyperemia and chemosis prediction method, in some cases the caruncular edema prediction method may use images that have been pre-processed in a different manner. For example, a pre-processed image may be used that has been cropped so that the image includes the caruncle and part of the iris. Alternatively, a pre-processed image may be used that has been cropped so that the image does not include the iris but does include the caruncle.

[0486]

[0487] 8. Eyelid redness prediction method

[0488] The eyelid redness prediction methods described herein may be performed by the server 20.

[0489] FIG. 26 is a flowchart showing a method for predicting eyelid redness.

[0490] Referring to FIG. 26, the server 20 acquires a facial image in step S400, preprocesses the acquired facial image in step S410, inputs the preprocessed image into the above-mentioned fourth prediction model (eyelid redness prediction model) in step S420, and acquires an output value of the fourth prediction model in step S430.

[0491] Acquiring face images

[0492] The server 20 acquires a face image in step S400. The server 20 may acquire the face image from the user terminal 10.

[0493] Facial image preprocessing

[0494] The server 20 may pre-process the facial image acquired in step S410. The server 20 may perform the iris segmentation, eye contour segmentation, masking, cropping, and resizing described above on the acquired facial image.

[0495] Segmentation Process

[0496] The server 20 may perform iris segmentation and eye contour segmentation, and the server 20 may therefore determine the pixels in the captured face image that correspond to the iris and the pixels that correspond to the inside of the eye contour. The server 20 may determine the coordinate values ​​of the pixels that correspond to the iris and the pixels that correspond to the inside of the eye contour.

[0497] However, in performing the eyelid redness prediction method, as will be described later, if a masking process is performed, then iris segmentation needs to be performed, but if a masking process is not performed, then it is permissible not to perform iris segmentation.

[0498] Masking Process

[0499] The server 20 may perform a second masking on the face image based on the determined information about the pixels. Through the second masking process, the server 20 may remove pixel values ​​of pixels corresponding to the externally exposed iris (cornea) among the pixels included in the face image. Thus, the pixel values ​​of pixels corresponding to the area excluding the iris (cornea) of the left eye and the iris (cornea) of the right eye may remain the original pixel values, but the pixel values ​​of pixels corresponding to the iris (or cornea) of the left eye and the iris (or cornea) of the right eye may be removed or changed to other values.

[0500] However, in performing the eyelid redness prediction method, it is advantageous in some aspects to perform pre-processing to mask the iris (cornea), although it is permissible not to perform masking on the iris.

[0501] Cropping

[0502] The server 20 may crop the masked face image. The server 20 may crop the masked face image to generate a left eye cropped image and a right eye cropped image. In performing the eyelid redness prediction method, the server 20 may use the third cropping (eyelid inclusion cropping) method among the cropping methods described above. The third cropping method has already been described in detail, so a detailed description thereof will be omitted here.

[0503] Resize and flip processing

[0504] The server 20 may resize the size of the left-eye cropped image and the size of the right-eye cropped image to a predetermined size.

[0505] On the other hand, if the fourth predictive model is not binarized but rather implemented as one model, the server 20 may horizontally flip either the left-eye cropped image or the right-eye cropped image as described above. The server 20 does not horizontally flip the other of the left-eye cropped image and the right-eye cropped image. Here, it has been determined that the criteria for determining whether one of the left-eye image and the right-eye image is to be horizontally flipped is the same as the criteria applied when the fourth predictive model is trained. That is, in training the fourth predictive model, if the left-eye image is flipped and the right-eye image is not flipped, the server 20 will flip the left-eye image and will not flip the right-eye image as well.

[0506] As described above, in implementing the fourth prediction model, if the fourth prediction model is binarized into a fourth left eye prediction model and a fourth right eye prediction model, the server 20 does not need to perform left-right inversion processing.

[0507] On the other hand, while it has been explained that when preprocessing is performed, segmentation, masking, cropping, resizing, and left-right flipping are performed, the order of these types of preprocessing may be changed within a range that allows the objective of the eyelid redness prediction method disclosed in the present application to be achieved.

[0508] Input preprocessed images

[0509] The server 20 may input the preprocessed image into a fourth predictive model in step S420.

[0510] If the fourth prediction model is not dualized and is realized as a single model, the server 20 sequentially inputs the right eye preprocessed image and the left eye preprocessed image that is flipped left and right into the fourth prediction model.

[0511] In implementing the fourth prediction model, when the fourth prediction model is binarized into a fourth left eye prediction model and a fourth right eye prediction model, the server 20 inputs the left eye pre-processed image into the fourth left eye prediction model, and the right eye pre-processed image into the fourth right eye prediction model. Alternatively, the server 20 may input the left eye pre-processed image into the fourth left eye prediction model, may input the left eye pre-processed image that has been flipped to the fourth right eye prediction model, may input the right eye pre-processed image into the fourth right eye prediction model, and may input the right eye pre-processed image that has been flipped to the fourth left eye prediction model.

[0512] In implementing the fourth prediction model, if the fourth prediction model is not binarized but implemented as one model, and is simultaneously trained to be able to determine whether or not there is eyelid redness without distinguishing between the left eye image and the right eye image, the server 20 may input the left eye preprocessed image and the right eye preprocessed image into the fourth prediction model without flipping them horizontally. Alternatively, the server 20 may input the left eye preprocessed image, the left eye preprocessed image that is flipped horizontally, the right eye preprocessed image, and the right eye preprocessed image that is flipped horizontally into the fourth prediction model.

[0513]

[0514] Eyelid redness prediction results

[0515] The server 20 may output a result value output from the fourth prediction model in step S430. The result value may be a predicted probability value for eyelid redness captured in the image. Based on a predefined threshold, if the predicted probability value is equal to or higher than the threshold, the server 20 determines that eyelid redness is present, or if the predicted probability value is less than the threshold, the server 20 determines that eyelid redness is not present.

[0516] The server 20 may obtain both the predicted results for the left eye and the predicted results for the right eye.

[0517] When the server 20 inputs the left eye preprocessed image into the fourth left eye prediction model, inputs the left eye preprocessed image that is left-right inverted into the fourth right eye prediction model, inputs the right eye preprocessed image into the fourth right eye prediction model, and inputs the right eye preprocessed image that is left-right inverted into the fourth left eye prediction model, the server 20 may obtain a prediction result for the left eye that takes into account both the result obtained by inputting the left eye preprocessed image into the fourth left eye prediction model and the result obtained by inputting the left eye preprocessed image that is left-right inverted into the fourth right eye prediction model. Here, the server 20 may obtain a prediction result for the right eye that takes into account both the result obtained by inputting the right eye preprocessed image into the fourth right eye prediction model and the result obtained by inputting the right eye preprocessed image that is left-right inverted into the fourth left eye prediction model.

[0518] For example, the server 20 may obtain a predicted result for the left eye based on whether an average value of a result obtained by inputting a left eye preprocessed image into a fourth left eye prediction model and a result obtained by inputting a left-right inverted left eye preprocessed image into a fourth right eye prediction model is equal to or higher than a threshold value.

[0519] As another example, if either the value of the result obtained by inputting the left eye preprocessed image into the fourth left eye prediction model, or the value of the result obtained by inputting the left eye inverted preprocessed image into the fourth right eye prediction model, is equal to or higher than the above-mentioned threshold value, the server 20 may predict that the left eye has eyelid redness.

[0520] As yet another example, if both the results obtained by inputting the left eye preprocessed image into a fourth left eye prediction model and the results obtained by inputting the left-right inverted left eye preprocessed image into a fourth right eye prediction model are equal to or higher than the above-mentioned threshold value, the server 20 may predict that the left eye has eyelid redness.

[0521] When the server 20 inputs the left eye preprocessed image, the left eye preprocessed image that is horizontally inverted, the right eye preprocessed image, and the right eye preprocessed image that is horizontally inverted into the fourth prediction model that is not binarized, the server 20 may obtain a prediction result for the left eye that takes into account both the result obtained by inputting the left eye preprocessed image into the fourth prediction model and the result obtained by inputting the left eye preprocessed image that is horizontally inverted into the fourth prediction model. Here, the server 20 may obtain a prediction result for the right eye that takes into account both the result obtained by inputting the right eye preprocessed image into the fourth prediction model and the result obtained by inputting the right eye preprocessed image that is horizontally inverted into the fourth prediction model.

[0522] For example, the server 20 may obtain a predicted result for the left eye based on whether an average value of a result obtained by inputting a left eye preprocessed image into the fourth prediction model and a result obtained by inputting a left-right inverted left eye preprocessed image into the fourth prediction model is equal to or higher than a threshold value.

[0523] As another example, if either the value of the result obtained by inputting the left eye pre-processed image into the fourth prediction model or the value of the result obtained by inputting the left eye pre-processed image that has been mirrored into the fourth prediction model is equal to or higher than the above-mentioned threshold value, the server 20 may predict that there is left eyelid redness.

[0524] As yet another example, if both the results obtained by inputting the left eye pre-processed image into the fourth prediction model and the results obtained by inputting the left eye pre-processed image flipped left eye into the fourth prediction model are equal to or higher than the above-mentioned threshold value, the server 20 may predict that there is left eyelid redness.

[0525] The above-described method may be similarly applied to determining whether or not there is right eyelid redness.

[0526]

[0527] 9. Eyelid edema prediction method

[0528] The eyelid edema prediction method described herein may be performed by the server 20.

[0529] FIG. 27 is a flowchart showing a method for predicting eyelid edema.

[0530] Referring to FIG. 27, the server 20 acquires a facial image in step S500, preprocesses the acquired facial image in step S510, inputs the preprocessed image into the above-mentioned fifth prediction model (eyelid edema prediction model) in step S520, and acquires an output value of the fifth prediction model in step S530.

[0531] The eyelid edema prediction method is the same as or very similar to the eyelid redness prediction method, except that the fifth prediction model is used instead of the fourth prediction model, and the final result value obtained is a prediction value regarding whether or not there is eyelid edema, and therefore a detailed description of the eyelid edema prediction method is omitted.

[0532]

[0533] As described above, the eyelid redness prediction method and the eyelid edema prediction method use the same image pre-processing method, but differ from each other only in the prediction models into which the pre-processed images are input. Thus, after the image pre-processing described above, the images may be input into different prediction models.

[0534]

[0535] 10. Method for predicting clinical activity score for thyroid eye disease

[0536] The method for predicting a clinical activity score for thyroid eye disease described in the present application is described below.

[0537] FIG. 28 is a diagram showing a method for predicting clinical activity score for thyroid eye disease.

[0538] The server 20 may acquire a facial image.

[0539] The server 20 performs two different types of pre-processing on one face image. The first pre-processing (hereinafter, the first pre-processing) includes iris segmentation, eye contour segmentation, first masking, second cropping (eye contour cropping), resizing, and left-right flipping, and the second pre-processing (hereinafter, the second pre-processing) includes iris segmentation, eye contour segmentation, second masking, third cropping (eyelid-containing cropping), resizing, and left-right flipping. However, as described in the eyelid redness prediction method, the iris segmentation and the second masking may be omitted.

[0540] The server 20 obtains a first preprocessed image by performing a first preprocessing on the acquired face image, and the first preprocessed image includes a first left-eye preprocessed image and a first right-eye preprocessed image. Here, either the first left-eye preprocessed image or the first right-eye preprocessed image is a left-right flipped image. Furthermore, as already described in detail, the first preprocessed image is an image obtained using a second cropping, in which the number of pixels corresponding to the eyelids in the first preprocessed image is minimized, and pixels corresponding to the conjunctiva and caruncle exposed to the outside are included. Furthermore, the first preprocessed image is an image obtained using a first masking, in which pixel values ​​of pixels corresponding to the iris (or cornea) and eyelids (upper eyelid and lower eyelid) are removed, but pixel values ​​of pixels corresponding to the conjunctiva and caruncle exposed to the outside remain.

[0541] Furthermore, the server 20 obtains a second preprocessed image by performing a second preprocessing on the obtained face image, and the second preprocessed image includes a second left-eye preprocessed image and a second right-eye preprocessed image. Here, either the second left-eye preprocessed image or the second right-eye preprocessed image is an image on which left-right flipping is performed. Furthermore, as already described in detail, the second preprocessed image is an image obtained by using a third cropping, so the second preprocessed image includes sufficient pixels corresponding to the eyelids. Furthermore, when a second masking method is used to obtain the second preprocessed image, pixel values ​​of pixels corresponding to the iris (or cornea) and eyelids (upper eyelid and lower eyelid) may be removed.

[0542] The server 20 sequentially inputs the first preprocessed images (the first left eye preprocessed image and the first right eye preprocessed image) into the first prediction model. The server 20 obtains a result value (probability value) of the first prediction model for the first left eye preprocessed image, and determines whether or not there is conjunctival hyperemia of the left eye based on the result value. Furthermore, the server 20 obtains a result value (probability value) of the first prediction model for the first right eye preprocessed image, and determines whether or not there is conjunctival hyperemia of the right eye based on the result value.

[0543] The server 20 combines the judgment result regarding the left eye and the judgment result regarding the right eye to finally judge whether or not conjunctival hyperemia is present in both eyes. For example, when it is judged that conjunctival hyperemia is present in either the left eye or the right eye, or both, the server 20 finally judges that conjunctival hyperemia is present.

[0544] Next, the server 20 inputs the first preprocessed images (the first left eye preprocessed image and the first right eye preprocessed image) into the second prediction model in sequence. The server 20 obtains a result value (probability value) of the second prediction model for the first left eye preprocessed image, and determines whether or not there is chemosis of the left eye based on the result value. Furthermore, the server 20 obtains a result value (probability value) of the second prediction model for the first right eye preprocessed image, and determines whether or not there is chemosis of the right eye based on the result value.

[0545] The server 20 combines the judgment result regarding the left eye and the judgment result regarding the right eye to finally judge whether or not chemosis is present in both eyes. For example, when it is judged that chemosis is present in either the left eye or the right eye or both, the server 20 finally judges that chemosis is present.

[0546] Next, the server 20 inputs the first preprocessed images (the first left eye preprocessed image and the first right eye preprocessed image) into the third prediction model in sequence. The server 20 obtains a result value (probability value) of the third prediction model for the first left eye preprocessed image, and determines whether or not there is lacrimal caruncle edema of the left eye based on the result value. Furthermore, the server 20 obtains a result value (probability value) of the third prediction model for the first right eye preprocessed image, and determines whether or not there is lacrimal caruncle edema of the right eye based on the result value.

[0547] The server 20 combines the judgment result for the left eye and the judgment result for the right eye to finally judge whether or not there is lacrimal caruncle edema in both eyes. For example, when it is judged that there is lacrimal caruncle edema in either or both of the left eye and the right eye, the server 20 finally judges that there is lacrimal caruncle edema.

[0548] The server 20 sequentially inputs the second preprocessed images (the second left-eye preprocessed image and the second right-eye preprocessed image) into the fourth prediction model. The server 20 obtains a result value (probability value) of the fourth prediction model for the second left-eye preprocessed image, and determines whether or not there is left-eyelid redness based on the result value. Furthermore, the server 20 obtains a result value (probability value) of the fourth prediction model for the second right-eye preprocessed image, and determines whether or not there is right-eyelid redness based on the result value.

[0549] The server 20 combines the judgment result for the left eye and the judgment result for the right eye to make a final judgment as to whether or not there is eyelid redness in both eyes. For example, if it is judged that there is eyelid redness in either or both of the left eye or the right eye, the server 20 finally judges that there is eyelid redness.

[0550] The server 20 sequentially inputs the second preprocessed images (the second left eye preprocessed image and the second right eye preprocessed image) into the fifth prediction model. The server 20 obtains a result value (probability value) of the fifth prediction model for the second left eye preprocessed image, and determines whether or not there is eyelid edema of the left eye based on the result value. Furthermore, the server 20 obtains a result value (probability value) of the fifth prediction model for the second right eye preprocessed image, and determines whether or not there is eyelid edema of the right eye based on the result value.

[0551] The server 20 combines the judgment result for the left eye and the judgment result for the right eye to make a final judgment as to whether or not there is eyelid edema in both eyes. For example, if it is judged that there is eyelid edema in either the left eye or the right eye, or both, the server 20 finally judges that there is eyelid edema.

[0552] If a symptom is determined to be present through the predictive models, the server 20 may provide a predefined score for the symptom (e.g., a score of 1). The server may provide scores for each of the five symptoms according to the determination results for the five predictive models, and may also obtain a value obtained by adding up all the scores.

[0553] The above-mentioned method for predicting a clinical activity score for thyroid eye disease described in the present application has been described as being executed by the server 20. However, the above-mentioned method may be executed by the user terminal 10. Alternatively, the pre-processing of the above-mentioned method may be executed by the user terminal 10, and the determination for each of the symptoms may be executed by the server. That is, the above-mentioned steps may be appropriately distributed and executed by the user terminal 10 and the server 20.

[0554]

[0555] 11. Recommendation of clinic visits based on continuous monitoring of clinical activity scores for thyroid eye disease

[0556] Below, we describe a method for serially monitoring clinical activity scores for thyroid eye disease and for recommending clinic visits based on the monitoring methods described herein.

[0557] FIG. 29 illustrates a method for serially monitoring clinical activity scores for thyroid eye disease and for recommending clinic visits based on the monitoring methods described herein.

[0558] The user terminal 10 may output, via the display 112, a guide for acquiring a face image in step S600.

[0559] The user terminal 10 may output an image captured in real time through the camera 140 (e.g., an image of the user's face) via the display 112. Here, a guide may be output together.

[0560] The user terminal 10 may acquire a facial image of the user's face via the camera 140 in step S610.

[0561] The user terminal 10 may transmit the acquired face image to the server 20 in step S620.

[0562] The user terminal 10 may output, through the display, a graphical user interface (GUI) for receiving user input regarding spontaneous retrobulbar pain and pain when attempting upward or downward gaze among a total of seven items considered in determining a clinical activity score for thyroid eye disease. Next, the user terminal 10 may receive the user's responses to these two items in step S630. The user terminal 10 may give a predetermined score (e.g., a score of 1) for each of the items based on the user's input responses. For example, if the user provides an input that the user has spontaneous retrobulbar pain, the user terminal 10 may give a score of 1 for that item. Furthermore, if the user provides an input that the user has pain when attempting upward or downward gaze, the user terminal 10 may give a score of 1 for that item.

[0563] In step S640, the user terminal 10 may receive from the server 20 the assessment results or the total scores regarding conjunctival redness, conjunctival swelling, caruncle swelling, eyelid redness, and eyelid swelling, among the total of seven items taken into consideration in determining the clinical activity score for thyroid eye disease based on the acquired facial image.

[0564] In step S650, the user terminal 10 may calculate a final clinical activity score for thyroid eye disease based on the score determined by user input and the score received from the server 20 or the score determined based on the determination result received from the server.

[0565] The user terminal 10 may store the time when the user's face image was acquired, or the time when the final calculated clinical activity score for thyroid eye disease was acquired, or a corresponding time (hereinafter referred to as measurement time, yy / mm / dd, hh:mm) together with the calculated clinical activity score in the memory 130. Alternatively, the user terminal 10 may transmit the above-mentioned measurement time and the corresponding clinical activity score to the server 20. Here, the server 20 may store the measurement time and the clinical activity score related to the user terminal 10 or the user in step S660.

[0566] On the other hand, the measurement time includes information about the date. The measurement time may include both information about the date and information about the hour and / or minute. Alternatively, the measurement time may only include information about the date and not information about the hour or minute.

[0567] In step S670, the user terminal 10 may output, via the display 112, information for recommending that the user visit a hospital for a detailed examination based on the calculated clinical activity score.

[0568] If the calculated clinical activity score is less than a score of 3, the user terminal 10 may output information via the display 112 indicating that there is no risk of thyroid eye disease.

[0569] If the calculated clinical activity score is a score of 3 or 4, the user terminal 10 may, at the option, output through the display 112 information indicating that there is no risk of thyroid eye disease, or may output through the display 112 information recommending that the user visit a hospital for a detailed examination.

[0570] If the calculated clinical activity score is equal to or higher than a score of 5, the user terminal 10 may output information via the display 112 to recommend that the user visit a hospital for a detailed examination.

[0571] If the calculated clinical activity score is a score of 3 or 4, the clinical activity score measured a predetermined period (e.g., one week) before the corresponding time point may be determined, and whether the clinical activity score was a score of 3 or 4 during the corresponding time interval (hereinafter, monitoring time interval) may be determined. Here, if a score of 3 or 4 occurs one or more times during the monitoring time interval, the user terminal 10 outputs, via the display 112, information to recommend that the user visit a hospital for a detailed examination. If a score of 3 or 4 never occurs during the monitoring time interval, the user terminal 10 outputs, via the display 112, information indicating that there is no risk of thyroid eye disease.

[0572] If the calculated clinical activity score is equal to or higher than a score of 3, the user terminal 10 may output information via the display 112 to recommend that the user visit a hospital for a detailed examination, without making any additional determinations regarding past records.

[0573] In the above description, when outputting information to a user through the user terminal 10, an example has been described in which the information is output visually through the display 112; however, in some cases, the information may be output audibly through a speaker.

[0574] Further, the method of continuously monitoring a clinical activity score for thyroid eye disease and the method of recommending a clinic visit based on the monitoring method described herein have been described as being performed by a user terminal 10. However, the steps of the above-described method may be suitably distributed and performed by the user terminal 10 and the server 20. For example, if the measurement time and the clinical activity score are transmitted to and stored in the server 20, it may be determined by the server 20 whether a score of 3 or 4 occurred during the monitoring time interval.

[0575]

[0576] 12. Experimental Example #1

[0577] [(1) Preparation of face image]

[0578] 1,020 facial images were prepared. Each facial image included both the left and right eyes and was acquired according to a predetermined imaging configuration.

[0579] [(2) Securing labeling information for facial images]

[0580] For each of the 1,020 facial images, information regarding conjunctival congestion, chemosis, caruncle edema, eyelid redness, and eyelid edema for the left eye, and information regarding conjunctival congestion, chemosis, caruncle edema, eyelid redness, and eyelid edema for the right eye was obtained, and this data was used as labeling data.

[0581] Among the 1,020 datasets, 714 datasets were used as training datasets, 102 datasets were used as validation sets, and 204 datasets were used as test sets.

[0582] Furthermore, splitting the 1,020 datasets into a training dataset, a validation set, and a test set was performed randomly 30 times, thus creating the first to thirtieth training dataset groups.

[0583] [(3) Obtaining a first preprocessed image and a second preprocessed image of a facial image]

[0584] For each of the 1,020 face images, a second cropping process (eye contour cropping) was performed for each of the left and right eyes in the above-mentioned manner to obtain a first left-eye preprocessed image and a first right-eye preprocessed image. Here, a left-right inverted image was used as the first right-eye preprocessed image, and a left-right non-inverted image was used as the first left-eye preprocessed image. Meanwhile, both the first left-eye preprocessed image and the first right-eye preprocessed image were images on which the above-mentioned first masking process was performed.

[0585] For each of the 1,020 face images, a third cropping process (eyelid-inclusive cropping) was performed for each of the left and right eyes in the above-mentioned manner to obtain a second left-eye preprocessed image and a second right-eye preprocessed image. Here, a left-right inverted image was used as the second right-eye preprocessed image, and a non-left-right inverted image was used as the second left-eye preprocessed image. Meanwhile, both the second left-eye preprocessed image and the second right-eye preprocessed image were images on which no masking process was performed.

[0586] [(4) Training of the first to fifth prediction models according to Experimental Example #1]

[0587] The secured first preprocessed image and the secured plurality of labeling information thereof, and the secured second preprocessed image and the secured plurality of labeling information thereof were used in training the first to fifth prediction models.

[0588] As the prediction model, a model using the above-mentioned ViT as the backbone architecture was used, and each of the prediction models was trained in an integrated state as one model without being divided into a left eye prediction model and a right eye prediction model.

[0589] [(5) Obtaining a prediction result for each symptom by using a prediction model]

[0590] The prediction results were obtained using the test dataset for the trained first to fifth prediction models, where the left-right flipped preprocessed image was used as the right-eye image and the left-right non-flip preprocessed image was used as the left-eye image.

[0591] [(6) Accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the eyelid redness prediction model according to Experimental Example #1]

[0592] The values ​​shown in Table 1 are the average accuracy, sensitivity, specificity, PPV, and NPV measured for the first through fifth predictive models trained on each of the 30 datasets according to Experimental Example #1 described above.

[0593] [Table 1]

[0594]

[0595] 13. Experimental Example #2

[0596] [(1) Preparation of face image]

[0597] The face images used in Experimental Example #1 were used as is.

[0598] [(2) Securing labeling information for facial images]

[0599] The multiple labeling information of the face images used in Experimental Example #1 was used as is.

[0600] [(3) Obtaining a first preprocessed image and a second preprocessed image of a facial image]

[0601] For each of the 1,020 face images, the second cropping process (eye contour cropping) was performed for each of the left and right eyes in the manner described above to obtain a first preprocessed image. Unlike Experimental Example #1, a first left eye preprocessed image that was not flipped horizontally, a first left eye preprocessed image that was flipped horizontally, a first right eye preprocessed image that was not flipped horizontally, and a first right eye preprocessed image that was flipped horizontally were obtained and used in training. Here, both the first left eye preprocessed image and the first right eye preprocessed image were images on which the first masking process described above was performed.

[0602] For each of the 1,020 face images, a third cropping process (eyelid-inclusive cropping) was performed for each of the left and right eyes in the manner described above to obtain second preprocessed images. Unlike Experimental Example #1, a second left eye preprocessed image that was not flipped horizontally, a second left eye preprocessed image that was flipped horizontally, a second right eye preprocessed image that was not flipped horizontally, and a second right eye preprocessed image that was flipped horizontally were obtained and used in training. Here, both the second left eye preprocessed image and the second right eye preprocessed image were images that were not subjected to masking processes.

[0603] [(4) Training of the first to fifth prediction models according to Experimental Example #2]

[0604] The secured first preprocessed image and the secured plurality of labeling information thereof, and the secured second preprocessed image and the secured plurality of labeling information thereof were used in training the first to fifth prediction models.

[0605] As the prediction model, a model using the above-mentioned ViT as a backbone architecture was used, and each prediction model was trained by dividing it into a left eye prediction model and a right eye prediction model. In particular, when the left eye prediction model was trained, a left eye preprocessed image that was not flipped left-right and a right eye preprocessed image that was flipped left-right were used, and when the right eye prediction model was trained, a right eye preprocessed image that was not flipped left-right and a left eye preprocessed image that was flipped left-right were used.

[0606] [(5) Obtaining a prediction result for each symptom by using a prediction model]

[0607] Using the test data set for the trained first to fifth prediction models, prediction results were obtained, where the prediction results for the right eye were obtained by inputting the non-flipped right-eye preprocessed image into each of the right eye prediction models, and the prediction results for the left eye were obtained by inputting the non-flipped left-eye preprocessed image into each of the left eye prediction models.

[0608] [(6) Accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the eyelid redness prediction model according to Experimental Example #2]

[0609] The values ​​shown in [Table 2] are the average accuracy, sensitivity, specificity, PPV, and NPV measured for the first through fifth predictive models trained on each of the 30 datasets according to Experimental Example #2 described above.

[0610] [Table 2]

[0611]

[0612] 1: System

[0613] 10: User terminal

[0614] 20: Server (Other possible items) [Item 1] preparing a conjunctival congestion prediction model, a conjunctival edema prediction model, a caruncular edema prediction model, an eyelid redness prediction model, and an eyelid edema prediction model; acquiring a facial image of a subject; obtaining a first processed image and a second processed image from the facial image, where the first processed image is different from the second processed image; applying the first processed image to the conjunctival hyperemia prediction model, the conjunctival chemosis prediction model, and the caruncular chemosis prediction model to obtain prediction values ​​for conjunctival hyperemia, conjunctival chemosis, and caruncular chemosis, respectively; applying the second processed image to the eyelid redness prediction model and the eyelid edema prediction model to obtain predictions for eyelid redness and eyelid edema, respectively; and determining the likelihood that the subject has thyroid eye disease based on the predictive values ​​for the conjunctival injection, the conjunctival edema, the caruncular edema, the eyelid redness, and the eyelid edema. Equipped with Here, the first processed image is an image in which an area corresponding to an inside of an iris contour and an area corresponding to an outside of an eye contour are masked, and the image is cropped along a first area including the eye contour based on position information of pixels corresponding to the iris contour included in the eye and position information of pixels corresponding to the eye contour; and Here, the second processed image is an image cropped along a second region larger than the first region based on position information of pixels corresponding to the contour of the iris included in the eye and position information of pixels corresponding to the contour of the eye. A computer-implemented method for predicting thyroid eye disease. [Item 2] 2. The computer-implemented method for predicting thyroid eye disease described in item 1, wherein the position information of the pixels corresponding to the contour of the iris included in the eye and the position information of the pixels corresponding to the contour of the eye are obtained by a segmentation model. [Item 3] the first processed image includes a first processed left-eye image and a first processed right-eye image; and The second processed image includes a second processed left-eye image and a second processed right-eye image. Item 1. A computer-implemented method for predicting thyroid eye disease according to item 1. [Item 4] The conjunctival hyperemia prediction model includes a left eye conjunctival hyperemia prediction model and a right eye conjunctival hyperemia prediction model, The chemotherapy prediction model includes a left eye chemotherapy prediction model and a right eye chemotherapy prediction model, The lacrimal caruncle edema prediction model includes a left eye lacrimal caruncle edema prediction model and a right eye lacrimal caruncle edema prediction model, The eyelid redness prediction model includes a left eyelid redness prediction model and a right eyelid redness prediction model, and The eyelid edema prediction model includes a left eyelid edema prediction model and a right eyelid edema prediction model. Item 3. A computer-implemented method for predicting thyroid eye disease according to item 3. [Item 5] the predicted value for conjunctival hyperemia is determined based on a result obtained by inputting the first processed left eye image into the left conjunctival hyperemia prediction model and a result obtained by inputting the first processed right eye image into the right conjunctival hyperemia prediction model; the predicted value for chemotherapy is determined based on results obtained by inputting the first processed left eye image into the left eye chemotherapy prediction model and results obtained by inputting the first processed right eye image into a right eye chemotherapy prediction model; the predicted value for the caruncular edema is determined based on a result obtained by inputting the first processed left eye image into the left eye caruncular edema prediction model and a result obtained by inputting the first processed right eye image into the right eye caruncular edema prediction model; the predicted value for eyelid redness is determined based on results obtained by inputting the second processed left eye image into the left eyelid redness prediction model and results obtained by inputting the second processed right eye image into the right eyelid redness prediction model; and the predicted value for eyelid edema is determined based on results obtained by inputting the second processed left eye image into the left eyelid edema prediction model and results obtained by inputting the second processed right eye image into the right eyelid edema prediction model. Item 5. A computer-implemented method for predicting thyroid eye disease according to item 4. [Item 6] processing one of the first processed left-eye image and the first processed right-eye image by mirror-flipping; and processing one of the second processed left-eye image and the second processed right-eye image by left-right flipping; 4. The computer-implemented method for predicting thyroid eye disease according to item 3, further comprising: [Item 7] the predicted value for the conjunctival hyperemia is determined based on results obtained by inputting the image flipped left-right and by inputting the image not flipped left-right into the conjunctival hyperemia prediction model; the predicted value for the chemosis is determined based on results obtained by inputting the image flipped and by inputting the image not flipped into the chemosis prediction model, the prediction value for the caruncular edema is determined based on results obtained by inputting the image flipped left-right and by inputting the image not flipped left-right into the caruncular edema prediction model, the predicted value for eyelid redness is determined based on results obtained by inputting the image flipped and by inputting the image not flipped into the eyelid redness prediction model; and The prediction value for the eyelid edema is determined based on results obtained by inputting the image flipped and by inputting the image not flipped into the eyelid edema prediction model. 7. A computer-implemented method for predicting thyroid eye disease according to item 6. [Item 8] resizing the first processed left-eye image and the first processed right-eye image; and Resizing the second processed left-eye image and the second processed right-eye image. 4. The computer-implemented method for predicting thyroid eye disease according to item 3, further comprising:

Claims

Step of receiving an evaluation of the subjective pain of a subject, wherein the evaluation of the subjective pain of the subject is obtained via an interface for obtaining the subjective pain of the subject, the step of receiving; Step of calculating a score for spontaneous retrobulbar pain and a score for pain when attempting upward or downward gaze based on a pre-stored score determination algorithm and the evaluation of the subjective pain of the subject; Step of receiving a face image representing at least one eye of the subject, wherein the face image is captured via an image sensor, the step of receiving; Step of obtaining a score for conjunctival congestion, a score for conjunctival edema, a score for lacrimal caruncle edema, a score for eyelid erythema, and a score for eyelid edema based on one or more sign prediction models and the face image, wherein the one or more sign prediction models are pre-trained using corresponding datasets, and each of the one datasets corresponding to the one sign prediction models has an image representing at least one of the conjunctival congestion, the conjunctival edema, the lacrimal caruncle edema, the eyelid erythema, and the eyelid edema, the step of obtaining; Step of calculating a final score by adding the scores of the spontaneous retrobulbar pain, the pain when attempting upward or downward gaze, the conjunctival congestion, the conjunctival edema, the lacrimal caruncle edema, the eyelid erythema, and the eyelid edema with the same weight; Comprising; A computer-implemented method for generating information related to thyroid eye disease.

2. The method further comprises the step of determining whether to recommend that the user visit a hospital by comparing the final score with a final criterion, The step of determining whether to recommend that the user visit includes determining to recommend a visit to the hospital when the final score is 3 or more. The computer-implemented method according to claim 1.

3. The method further comprises the step of determining whether to recommend that the user visit a hospital by comparing the final score with a final criterion, The step of determining whether to recommend that the user visit is Determining to recommend a visit to the hospital when the final score is 5 or more, and Determining to recommend a visit to the hospital when the final score is 3 or 4 and at least one of the final scores in a predetermined period is 3 or more, Including, The computer-implemented method according to claim 1. **Claim 4**: The predetermined period is set as one week slightly before the day when the final score is calculated. The computer-implemented method according to claim 3. **Claim 5**: The score of the conjunctival congestion is calculated based on the face image, the first prediction model, and the first evaluation criterion. The score of the conjunctival edema is calculated based on the face image, the second prediction model, and the second evaluation criterion. The score of the lacrimal caruncle edema is calculated based on the face image, the third prediction model, and the third evaluation criterion. The score of the eyelid flushing is calculated based on the face image, the fourth prediction model, and the fourth evaluation criterion. The score of the eyelid edema is calculated based on the face image, the fifth prediction model, and the fifth evaluation criterion. The computer-implemented method according to claim 1. **Claim 6**: The first evaluation criterion is satisfied when the probability value of the first prediction model is greater than the first threshold. The second evaluation criterion is satisfied when the probability value of the second prediction model is greater than the second threshold. The third evaluation criterion is satisfied when the probability value of the third prediction model is greater than the third threshold. The fourth evaluation criterion is satisfied when the probability value of the fourth prediction model is greater than the fourth threshold. The fifth evaluation criterion is satisfied when the probability value of the fifth prediction model is greater than the fifth threshold. The computer-implemented method according to claim 5. **Claim 7**: At least one of the first threshold, the second threshold, the third threshold, the fourth threshold, and the fifth threshold is different from another one of the first threshold, the second threshold, the third threshold, the fourth threshold, and the fifth threshold. The computer-implemented method according to claim 6. **Claim 8**: The score of the conjunctival congestion is 1) when the probability value of the first prediction model by inputting the left-eye image of the face image is greater than the first threshold, and / or 2) when the probability value of the first prediction model by inputting the right-eye image of the face image is greater than the first threshold is determined as 1. The score of the conjunctival edema is 1) when the probability value of the second prediction model by inputting the left-eye image of the face image is greater than the second threshold, and / or 2) when the probability value of the second prediction model by inputting the right-eye image of the face image is greater than the second threshold is determined as 1. The score of the lacrimal caruncle edema is 1) The probability value of the third prediction model by inputting the left eye image of the face image is greater than the third threshold value, and / or 2) The probability value of the third prediction model by inputting the right eye image of the face image is greater than the third threshold value, it is determined as 1 in case of; the score of the eyelid flushing is 1) The probability value of the fourth prediction model by inputting the left eye image of the face image is greater than the fourth threshold value, and / or 2) The probability value of the fourth prediction model by inputting the right eye image of the face image is greater than the fourth threshold value it is determined as 1 in case of, and the score of the eyelid swelling is 1) The probability value of the fifth prediction model by inputting the left eye image of the face image is greater than the fifth threshold value, and / or 2) The probability value of the fifth prediction model by inputting the right eye image of the face image is greater than the fifth threshold value it is determined as 1 in case of The computer-implemented method according to claim 6.

9. The evaluation of the subjective pain of the subject is received via a user terminal, the user terminal outputs the interface for obtaining the subjective pain of the subject, The computer-implemented method according to claim 1.

10. In the interface, a first question about whether the spontaneous pain behind the eyeball is felt and a second question about whether the pain is felt when attempting upward or downward gaze are displayed on the user terminal, The computer-implemented method according to claim 9.

11. In the interface, a first interface for receiving user input regarding the first question and a second interface for receiving user input regarding the second question are also displayed on the user terminal, The computer-implemented method according to claim 10.

12. When the user input regarding the first question indicates the presence of pain, the score of the spontaneous pain behind the eyeball is determined as 1, when the user input regarding the second question indicates the presence of pain, the score of the pain when attempting upward or downward gaze is determined as 1, The computer-implemented method according to claim 11.

13. The computer-implemented method according to claim 9, wherein the image sensor is integrated with the user terminal.

14. A computing device, a processor, A memory operatively coupled to the processor, the memory storing instructions executable by the processor to cause the computing device to perform the following operations: comprising: The operations are: Receiving an assessment of the subject's subjective pain, wherein the assessment of the subject's subjective pain is obtained via an interface for obtaining the subject's subjective pain; Calculating a score for spontaneous postorbital pain and a score for pain when attempting upward or downward gaze based on a pre-stored score determination algorithm and the assessment of the subject's subjective pain; Receiving a face image representing at least one eye of the subject, wherein the face image is captured via an image sensor; Obtaining a score for conjunctival hyperemia, a score for conjunctival edema, a score for caruncle edema, a score for eyelid erythema, and a score for eyelid edema based on one or more sign prediction models and the face image, wherein the one or more sign prediction models are pre-trained using corresponding datasets, and each of the one datasets corresponding to the one sign prediction models has an image representing at least one of the conjunctival hyperemia, the conjunctival edema, the caruncle edema, the eyelid erythema, and the eyelid edema; Calculating a final score by adding the scores of the spontaneous postorbital pain, the pain when attempting upward or downward gaze, the conjunctival hyperemia, the conjunctival edema, the caruncle edema, the eyelid erythema, and the eyelid edema with the same weight; comprising: A computing device. **Claim 15**: The score for the conjunctival hyperemia is calculated based on the face image, a first prediction model, and a first evaluation criterion. The score for the conjunctival edema is calculated based on the face image, a second prediction model, and a second evaluation criterion. The score for the caruncle edema is calculated based on the face image, a third prediction model, and a third evaluation criterion. The score for the eyelid erythema is calculated based on the face image, a fourth prediction model, and a fourth evaluation criterion. The score for the eyelid edema is calculated based on the face image, a fifth prediction model, and a fifth evaluation criterion. The computing device according to claim 14. **Claim 16**: The score for the conjunctival hyperemia is 1) the probability value of the first prediction model by inputting the left eye image of the face image is greater than a first threshold value, and / or 2) When the probability value of the first prediction model by inputting the right eye image of the face image is greater than the first threshold value it is determined as 1, and the score of the conjunctival edema is 1) When the probability value of the second prediction model by inputting the left eye image of the face image is greater than the second threshold value, and / or 2) When the probability value of the second prediction model by inputting the right eye image of the face image is greater than the second threshold value it is determined as 1, and the score of the caruncle edema is 1) When the probability value of the third prediction model by inputting the left eye image of the face image is greater than the third threshold value, and / or 2) When the probability value of the third prediction model by inputting the right eye image of the face image is greater than the third threshold value it is determined as 1, and the score of the eyelid flushing is 1) When the probability value of the fourth prediction model by inputting the left eye image of the face image is greater than the fourth threshold value, and / or 2) When the probability value of the fourth prediction model by inputting the right eye image of the face image is greater than the fourth threshold value it is determined as 1, and the score of the eyelid edema is 1) When the probability value of the fifth prediction model by inputting the left eye image of the face image is greater than the fifth threshold value, and / or 2) When the probability value of the fifth prediction model by inputting the right eye image of the face image is greater than the fifth threshold value it is determined as 1 The computing device according to claim 15.

17. The evaluation of the subjective pain of the subject is received via a user terminal, and the user terminal outputs the interface for acquiring the subjective pain of the subject. The computing device according to claim 14.

18. On the first screen, a first question about whether the spontaneous retrobulbar pain is felt and a second question about whether the pain is felt when attempting to look upward or downward are displayed on the user terminal, and a first interface for receiving a user input regarding the first question and a second interface for receiving a user input regarding the second question are also displayed on the user terminal. The computing device according to claim 17.

19. When the user input regarding the first question indicates the presence of pain, the score of the spontaneous retrobulbar pain is determined as 1. If the user input regarding the second question indicates the presence of pain, the score of the pain when attempting the upward or downward gaze is determined to be 1. The computing device according to claim 18.

20. When executed by one or more processors of a computing device, Receiving an evaluation of the subjective pain of a subject, wherein the evaluation of the subjective pain of the subject is obtained via an interface for obtaining the subjective pain of the subject; Calculating a score for spontaneous retrobulbar pain and a score for pain when attempting upward or downward gaze based on a pre-stored score determination algorithm and the evaluation of the subjective pain of the subject; Receiving a face image representing at least one eye of the subject, wherein the face image is captured via an image sensor; Obtaining a score for conjunctival hyperemia, a score for conjunctival edema, a score for lacrimal caruncle edema, a score for eyelid erythema, and a score for eyelid edema based on one or more sign prediction models and the face image, wherein the one or more sign prediction models are pre-trained using corresponding datasets, and each of the one datasets corresponding to the one sign prediction models has an image representing at least one of the conjunctival hyperemia, the conjunctival edema, the lacrimal caruncle edema, the eyelid erythema, and the eyelid edema; Calculating a final score by adding the scores of the spontaneous retrobulbar pain, the pain when attempting upward or downward gaze, the conjunctival hyperemia, the conjunctival edema, the lacrimal caruncle edema, the eyelid erythema, and the eyelid edema with the same weight; A computer-executable instruction that causes the computing device to perform operations including: A computer program.

Images