Disease diagnosis method using training model and system for carrying out same

Abstract

A symptom prediction method is proposed, comprising the steps of acquiring a facial image including an eye region, acquiring a plurality of result values ​​using the facial image and a diagnostic model for predicting an eye-related symptom, where the plurality of result values ​​include a plurality of predicted values ​​and a single consensus predicted value, and determining the presence of the eye-related symptom based on the consensus predicted value among the plurality of result values, where the diagnostic model is generated using training data in which a plurality of decision values ​​independently determined by two or more different diagnosticians regarding the presence of the eye-related symptom in the same diagnostic subject and a single consensus value determined by the diagnosticians through mutual agreement regarding the presence of the eye-related symptom in the same diagnostic subject are multi-labeled for one diagnostic image, where the plurality of predicted values ​​correspond to the plurality of decision values ​​and the consensus predicted value corresponds to the consensus value.

Description

[Technical field]

[0001] The present disclosure relates to a method for diagnosing disease using a training model and a system for implementing the same. [Background technology]

[0002] In the medical field, there has been active development in training models used for disease diagnosis.

[0003] Due to the characteristics of disease diagnosis, even for the same diagnostic subject, diagnostic results may vary depending on the experience or inclinations of the medical provider.

[0004] Therefore, in generating a training model for disease diagnosis, it is a well-known fact that if the diagnostic model is trained using only information about the diagnosis subject and information about the diagnosis result as training data without distinguishing medical providers, the different empirical rules or tendencies of medical providers to apply to the diagnosis result will be diluted during the training process, and the accuracy of the trained model will be reduced.

[0005] Therefore, in the field of disease diagnosis using training models, systems have been developed that, rather than generating diagnostic models without distinguishing between healthcare providers, generate diagnostic models for each healthcare provider and, if necessary, select a value determined by a majority vote of the resulting values ​​of each diagnostic model as the final predicted disease diagnosis value.

[0006] However, the accuracy of an artificial intelligence model cannot be 100%, and a system for predicting disease diagnosis values ​​using the result values ​​of each diagnostic model cannot guarantee the accuracy of the final disease diagnosis because many inaccurate predicted values ​​are used and inaccuracies accumulate.

[0007] Therefore, it is necessary to develop a method to obtain a final diagnostic value with reduced accuracy loss and to apply diagnostic results determined by multiple healthcare providers individually. Summary of the Invention [Problem to be solved by the invention]

[0008] The disclosure herein is directed to providing a training model that can reduce accuracy loss while taking into account the decision value for each diagnostician.

[0009] Additionally, the disclosure herein relates to providing training models capable of predicting disease by using images acquired using digital cameras available to the general public, rather than specialized medical diagnostic devices.

[0010] Additionally, the disclosure herein relates to providing a system that allows laypeople to monitor clinical activity scores for thyroid eye disease without the assistance of a physician and without an in-person visit to a hospital.

[0011] The technical problems to be solved by the present application are not limited to the above-mentioned technical problems, and other technical problems not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings. [Means for solving the problem]

[0012] According to an embodiment of the present application, there is provided a symptom prediction method comprising: acquiring a facial image including an eye region; acquiring a plurality of result values ​​using the facial image and a diagnostic model for predicting an eye-related symptom, where the plurality of result values ​​include a plurality of predicted values ​​and a single consensus predicted value; and determining the presence of the eye-related symptom based on the consensus predicted value among the plurality of result values, where the diagnostic model is generated using training data in which a plurality of decision values ​​independently determined by two or more different diagnosticians regarding the presence of the eye-related symptom in a same diagnostic subject and a single consensus value determined by the diagnosticians through mutual agreement regarding the presence of the eye-related symptom in the same diagnostic subject are multi-labeled for one diagnostic image, where the plurality of predicted values ​​correspond to the plurality of decision values ​​and the consensus predicted value corresponds to the consensus value, where the eye-related symptom is at least one of conjunctival redness, conjunctival edema, caruncle edema, eyelid redness, and eyelid edema.

[0013] According to an embodiment of the present application, at least a portion of the training data may have a consensus value that corresponds to a minority of the plurality of decision values.

[0014] According to an embodiment of the present application, when training a diagnostic model, the loss weight for the node where the consensus prediction value is output may be set higher than the loss weight for each node where a prediction value is output.

[0015] According to an embodiment of the present application, the step of obtaining a plurality of result values ​​may include a step of performing pre-processing on the face image for the eye region; and a step of obtaining a plurality of result values ​​using the pre-processed image and a diagnostic model.

[0016] According to an embodiment of the present application, the step of performing pre-processing may include: performing pre-processing on the facial image to mask the pupil and skin when the eye-related condition is at least one of conjunctival redness, conjunctival edema, and caruncular edema; and performing pre-processing on the facial image to mask the eye area when the eye-related condition is at least one of eyelid redness and eyelid edema.

[0017] According to an embodiment of the present application, the diagnosticians may be three ophthalmologists with over 15 years of experience, where the decision value may include three values ​​corresponding to each diagnostician.

[0018] According to an embodiment of the present application, at least one of the decision value and the agreement value may be a value determined by a diagnostician in an actual encounter with a subject regarding the presence of an eye-related condition.

[0019] According to an embodiment of the present application, at least one of the judgment value and the agreement value may be a value determined by a diagnostician through a facial image of a subject with respect to the presence of an eye-related condition.

[0020] According to an embodiment of the present application, a consensus value may be determined after a diagnostician determines each of the decision values.

[0021] According to an embodiment of the present application, the diagnostic model may be generated using at least one algorithm selected from the group of Convolutional Neural Network (CNN), Vision Transformer, Support Vector Machine (SVM), Random Forest, Gradient Boosting Algorithm, Artificial Neural Network (ANN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), ResNet, VGG, GoogLeNet and MobileNet.

[0022] However, the solutions to the problems of the present disclosure are not limited to the solutions described above, and a person skilled in the art will clearly understand other solutions not mentioned from this specification and the accompanying drawings. Beneficial Effects

[0023] According to the disclosure in the present application, by using training data in which multiple diagnostician's judgment values ​​and diagnostician consensus values ​​are multi-labeled for diagnostic images, a diagnostic model can be generated that is capable of reducing accuracy loss while taking into account the respective judgment values ​​of multiple diagnosticians.

[0024] According to the disclosure in the present application, by using training data in which diagnostic images are labeled with label values ​​determined by applying the judgment values ​​of multiple diagnosticians to the diagnostician consensus value, a diagnostic model can be generated that can reduce accuracy loss while taking into account the respective judgment values ​​of multiple diagnosticians.

[0025] According to the disclosure herein, diagnostic models can be generated that can predict the presence of disease by using images acquired using digital cameras available to the general public, rather than specialized medical diagnostic devices.

[0026] The disclosure herein provides a system that allows laypeople to monitor their clinical activity scores for thyroid eye disease without the assistance of a physician and without an in-person visit to a hospital. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 illustrates a system for diagnosing a disease, according to an embodiment.

[0028] [Diagram 2] FIG. 2 is a block diagram showing a configuration of a user device according to an embodiment.

[0029] [Diagram 3] FIG. 2 is a diagram illustrating a configuration of a server according to an embodiment.

[0030] [Figure 4] FIG. 1 illustrates a process for obtaining a decision value for each diagnostician, according to an embodiment.

[0031] [Diagram 5] FIG. 1 illustrates a process for obtaining consensus values ​​from multiple diagnosticians, according to an embodiment.

[0032] [Figure 6] FIG. 2 illustrates training data used to train a diagnostic model, according to an embodiment.

[0033] [Figure 7] FIG. 1 illustrates a process of predicting disease by using a trained diagnostic model, according to an embodiment.

[0034] [Figure 8] FIG. 2 illustrates training data used to train a diagnostic model, according to an embodiment.

[0035] [Figure 9] FIG. 1 illustrates a process of predicting disease by using a trained diagnostic model, according to an embodiment.

[0036] [Figure 10] FIG. 2 illustrates training data used to train a diagnostic model, according to an embodiment.

[0037] [Figure 11] FIG. 1 illustrates a process of predicting disease by using a trained diagnostic model, according to an embodiment.

[0038] [Figure 12] FIG. 1 illustrates a system for predicting clinical activity score for thyroid eye disease, according to an embodiment.

[0039] [Figure 13] FIG. 2 illustrates an image pre-processing process, according to an embodiment. [Figure 14] FIG. 2 illustrates an image pre-processing process, according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] Since the embodiments described in this specification are intended to clearly explain the concept of the present disclosure to those skilled in the art to which the present disclosure pertains, the present disclosure is not limited to the embodiments described in this specification, and the scope of the present disclosure should be interpreted as including modifications or variations within the concept of the present disclosure.

[0041] As terms used in this specification, currently widely used general terms are used in consideration of the functions in this disclosure. However, terms may vary according to the intentions, practices, or emergence of new technologies of those skilled in the art. However, when a specific term defined as having an optional meaning is used, the meaning of the term is explained. Therefore, the terms used in this specification should be interpreted based on the actual meaning and details of the terms throughout this specification, not simply the name of the term.

[0042] Numbers used in the description herein (eg, first, second, etc.) are merely identifiers to distinguish one element from another.

[0043] In the following embodiments, unless the context clearly indicates otherwise, expressions used in the singular form include expressions in the plural form.

[0044] In the following embodiments, terms such as "comprise" or "have" are intended to indicate the presence of features or elements disclosed in the specification, and are not intended to exclude the possibility that one or more other features or elements may be added.

[0045] The drawings attached to this specification are intended to easily explain the present disclosure, and the shapes shown in the drawings may be exaggerated to facilitate understanding of the present disclosure, so the present disclosure is not limited by the drawings.

[0046] If a particular embodiment is implemented in an alternative manner, certain processes may be performed out of the order described. For example, two processes described in succession may be performed substantially simultaneously or may be performed in the reverse order from that described.

[0047] In this specification, if a detailed description of a known configuration or function related to the present disclosure is determined to make the subject matter of the present disclosure unclear, the detailed description will be omitted.

[0048] According to an embodiment of the present disclosure, there is provided a symptom prediction method comprising: acquiring a facial image including an eye region; acquiring a plurality of result values ​​using the facial image and a diagnostic model for predicting an eye-related symptom, where the plurality of result values ​​include a plurality of predicted values ​​and a single consensus predicted value; and determining the presence of the eye-related symptom based on the consensus predicted value among the plurality of result values, where the diagnostic model is generated using training data in which a plurality of decision values ​​independently determined by two or more different diagnosticians regarding the presence of the eye-related symptom in a same diagnostic subject and a single consensus value determined by the diagnosticians through mutual agreement regarding the presence of the eye-related symptom in the same diagnostic subject are multi-labeled for one diagnostic image, where the plurality of predicted values ​​correspond to the plurality of decision values ​​and the consensus predicted value corresponds to the consensus value, where the eye-related symptom is at least one of conjunctival redness, conjunctival edema, caruncular edema, eyelid redness, and eyelid edema.

[0049] According to an embodiment of the present application, at least a portion of the training data may have a consensus value that corresponds to a minority of the plurality of decision values.

[0050] According to an embodiment of the present application, when training a diagnostic model, the loss weight for the node where the consensus prediction value is output may be set higher than the loss weight for each node where a prediction value is output.

[0051] According to an embodiment of the present application, the step of obtaining a plurality of result values ​​may include a step of performing pre-processing on the face image for the eye region; and a step of obtaining a plurality of result values ​​using the pre-processed image and a diagnostic model.

[0052] According to an embodiment of the present application, the step of performing pre-processing may include: performing pre-processing on the facial image to mask the pupil and skin when the eye-related condition is at least one of conjunctival redness, conjunctival edema, and caruncular edema; and performing pre-processing on the facial image to mask the eye area when the eye-related condition is at least one of eyelid redness and eyelid edema.

[0053] According to an embodiment of the present application, the diagnosticians may be three ophthalmologists with over 15 years of experience, where the decision value may include three values ​​corresponding to each diagnostician.

[0054] According to an embodiment of the present application, at least one of the decision value and the agreement value may be a value determined by a diagnostician in an actual encounter with a subject regarding the presence of an eye-related condition.

[0055] According to an embodiment of the present application, at least one of the judgment value and the agreement value may be a value determined by a diagnostician through a facial image of a subject with respect to the presence of an eye-related condition.

[0056] According to an embodiment of the present application, a consensus value may be determined after a diagnostician determines each of the decision values.

[0057] According to an embodiment of the present application, the diagnostic model may be generated using at least one algorithm selected from the group of Convolutional Neural Network (CNN), Vision Transformer, Support Vector Machine (SVM), Random Forest, Gradient Boosting Algorithm, Artificial Neural Network (ANN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), ResNet, VGG, GoogLeNet and MobileNet.

[0058] A diagnostic system and method for generating a diagnostic model according to an embodiment will be described below.

[0059] 1. Configuration of diagnostic system 10

[0060] FIG. 1 is a diagram illustrating a diagnostic system 10 for diagnosing a disease, according to an embodiment.

[0061] The diagnostic system 10 according to the embodiment may perform disease diagnosis using information acquired from a human subject. The information acquired from a human subject may be of various types. For example, the information acquired from a human subject may be of various types, such as images of all or part of the human body, sounds generated by the human, biosignals of the human, information about the biological responses of the human, and information about the behavior of the human.

[0062] Thus, for example, the diagnostic system 10 may predict the presence of a disease using images acquired from at least a portion of a user's body and a trained diagnostic model. As another example, the diagnostic system 10 may predict the presence of a disease using sounds generated from the user's body and a trained diagnostic model. As a specific example, but not limited to, the diagnostic system 10 may predict the presence of an eye disease using facial images acquired from a user's face and a trained eye disease diagnostic model.

[0063] For ease of explanation, the diagnostic system 10 will be described below assuming that a disease is diagnosed based on an image.

[0064] With reference to FIG. 1, a diagnostic system 10 may include a user device 100 and a server 200 .

[0065] The user device 100 is a device for directly and / or indirectly interacting with a user. The user device 100 may generate a diagnostic image by capturing an image of a diagnostic target that is at least a part of the user's body.

[0066] The user device 100 may transmit diagnostic images of the diagnosis target to the server 200, which will be described later. Specifically, the user device 100 may transmit the diagnostic images to the server 200 through wired and / or wireless data communication methods, or may transmit the diagnostic images to the server 200 through other external devices.

[0067] The user device 100 may receive information regarding the diagnosis results from the server 200. The user device 100 may provide the information obtained from the server 200 to the user in a visual and / or audio manner.

[0068] For example, user device 100 may include user input devices and / or capture devices such as smartphones, tablet computers, desktop and laptop computers, and digital cameras.

[0069] The server 200 may use the trained diagnostic model to predict the presence of a disease and / or condition in the diagnostic image. The diagnostic image may be received directly from the user device 100 or may be received through an external device, such as a relay device and / or another external server.

[0070] The server 200 may transmit the prediction results to the user device 100 and / or other external devices.

[0071] The server 200 may obtain a trained diagnostic model from an external device and store the trained diagnostic model. Without being limited thereto, the server 200 may generate a diagnostic model using diagnostic images labeled with label values ​​as training data.

[0072] Meanwhile, although the server 200 and the user device 100 have been described as being distinct from one another, the server 200 and the user device 100 may be implemented as a single device.

[0073] 2. Configuration of User Device 100

[0074] The configuration of the user device 100 in the configuration of the diagnostic system 10 will be described below.

[0075] FIG. 2 is a block diagram showing a configuration of a user device 100 according to an embodiment.

[0076] Referring to FIG. 2, the user device 100 may include a camera 110, a communication device 120, a user interface 130, a memory 140, and a processor 150.

[0077] The camera 110 is a digital camera and may include an image sensor and an image processor. The image sensor is a device for converting an optical image into an electrical signal and may be provided as a chip incorporating a number of photodiodes. Examples of the image sensor may include a charge-coupled device (CCD) and a complementary metal-oxide semiconductor (CMOS). Meanwhile, the image processor may perform image processing on the captured result and generate image information.

[0078] The communication device 120 may transmit or receive data and / or information to or from the outside world through wired and / or wireless communication. The communication device 120 may perform two-way or one-way communication.

[0079] The communication device 120 may include a wireless communication module and / or a wired communication module. In this specification, examples of the wireless communication module include a Wi-Fi (Registered Trademark) It may include a communication module and a cellular communication module.

[0080] The user interface 130 may output various types of information according to control commands from the processor 150. According to an embodiment, the user interface 130 may include a display for visually outputting information to the user. The user interface 130 may include a speaker for audibly outputting information to the user. The user interface 130 may include a vibration motor for tactilely outputting information to the user.

[0081] Meanwhile, the user device 100 may receive various types of information necessary for the operation of the user device 100 from the user through the user interface 130. For example, the user device 100 may receive, but is not limited to, information related to diseases and / or symptoms from the user through the user interface 130.

[0082] Memory 140 may store various processing programs, parameters for executing the processing of the programs, or data resulting from such processing. For example, memory 140 may store instructions for the operation of processor 150, which will be described later, various pre-processing algorithms for diagnostic images, and / or executable code for implementing training models. Additionally, memory 140 may store diagnostic images acquired through camera 110 and pre-processed diagnostic images.

[0083] The memory 140 may be implemented as a non-volatile semiconductor memory, a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), flash memory, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or other type of tangible non-volatile recording medium.

[0084] The processor 150 may control the overall operation of the user device 100 and may operate according to instructions stored in the memory 140 .

[0085] In particular, the processor 150 may control the camera 110 to capture diagnostic images and store the diagnostic images in the memory 140. Examples of diagnostic images may include any image related to a patient, such as a portion of the patient's body, a facial image, an eye image, and a full body image.

[0086] The processor 150 may process the diagnostic images using pre-processing algorithms and / or training models stored in the memory 140. The processor 150 may use different pre-processing algorithms and / or training models depending on the diagnostic subject, the type of disease, and the type of symptom. For example, when the symptom is eye-related, the processor 150 may perform pre-processing such as cropping and / or masking a portion of the image to make the eye portion more clearly visible. The processor 150 may perform various types of pre-processing such as, but not limited to, color correction and / or brightness correction of the image.

[0087] The processor 150 may transmit the image and / or the pre-processed image to the server 200 and / or an external device through the communication device 120.

[0088] The processor 150 processes the information received from the server 200 and / or the external device as follows: User Interface The information may be provided to the user via 130.

[0089] The processor 150 may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a state machine, an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), and combinations thereof.

[0090] 3. Server 200 Configuration

[0091] The configuration of the server 200 in the configuration of the diagnostic system 10 will be described below.

[0092] FIG. 3 is a diagram showing a configuration of the server 200 according to the embodiment.

[0093] Referring to FIG. 3, the server 200 may include a communication device 210, a memory 220, and a processor 230.

[0094] The communication device 210 may transmit and / or receive data and / or information to and from the outside world through wired and / or wireless communication. The communication device 210 may perform two-way or one-way communication.

[0095] The communication device 210 may include a wireless communication module and / or a wired communication module. In this specification, examples of a wireless communication module may include a Wi-Fi communication module and a cellular communication module.

[0096] The memory 220 may store various processing programs, parameters for executing the processing of the programs, parameters for executing such processing, or data resulting from such processing. For example, the memory 220 may store instructions for the operation of the processor 230 described below, various pre-processing algorithms for diagnostic images, and / or executable code for implementing training models. Examples of pre-processing algorithms and / or training models may include various pre-processing algorithms and / or training models depending on the diagnostic subject and / or disease.

[0097] The memory 220 may store diagnostic images received from the user device 100 and / or an external device. The received diagnostic images may be pre-processed images.

[0098] Examples of training models stored in memory 220 may include diagnostic models for predicting the presence of diseases and / or symptoms in diagnostic images. Memory 220 may include various types of diagnostic models for each diagnostic subject and / or diagnostic disease. Furthermore, memory 220 may store various types of diagnostic models for the same disease. For example, multiple diagnostic models for diagnosing the presence of conjunctival redness may be stored. Each diagnostic model may be a diagnostic model generated using a different algorithm and / or training data. Diagnostic models stored in memory 220 may be diagnostic models received from outside or may be diagnostic models generated by server 200.

[0099] The memory 220 may store training data and model generation algorithms used to generate the diagnostic models, specific details of which are described below.

[0100] The memory 220 may be implemented as a non-volatile semiconductor memory, a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), flash memory, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or other type of tangible non-volatile recording medium.

[0101] The processor 230 may control the overall operation of the server 200 and may operate according to instructions stored in the memory 220 .

[0102] The processor 230 may use the diagnostic models stored in the memory 220 to predict whether a diagnostic subject shown in a diagnostic image has a disease and / or condition.

[0103] The processor 230 may apply different diagnostic models to one diagnostic image, thereby generating various types of predicted results. For example, the processor 230 may use respective diagnostic models for diagnosing conjunctival redness, chemosis, eyelid redness, eyelid edema, and caruncular edema for one eye image to generate various types of predicted results, such as the presence of conjunctival redness, the presence of chemosis, the presence of eyelid redness, the presence of eyelid edema, and the presence of caruncular edema.

[0104] Without being limited thereto, the processor 230 may apply one diagnostic model for predicting multiple types of diseases and / or symptoms for one diagnostic image, thereby generating various types of predicted results. For example, the processor 230 may use one diagnostic model for diagnosing at least two selected from the group of conjunctival redness, chemosis, eyelid redness, eyelid edema, and caruncular edema for one eye image, and may generate at least two predicted results of the presence of conjunctival redness, the presence of chemosis, the presence of eyelid redness, the presence of eyelid edema, and the presence of caruncular edema.

[0105] The processor 230 may determine a diagnostic model to be used for prediction based on the diagnostic image. For example, when the processor 230 determines that the diagnostic image is a face image, the processor 230 may determine a diagnostic model for predicting diseases and / or conditions related to the face and may predict the diseases and / or conditions using the determined diagnostic model.

[0106] The processor 230 may perform pre-processing on the diagnostic image before using the diagnostic model. The processor 230 may perform pre-processing on the diagnostic image by using different pre-processing algorithms depending on the diagnostic subject and / or the diagnostic disease. For example, when the diagnostic subject and / or the diagnostic disease relates to the user's face, the processor 230 may crop the diagnostic image to leave only the face region. When the diagnostic subject and / or the diagnostic disease relates to the user's eye, the processor 230 may crop the diagnostic image to leave only the eye region.

[0107] The processor 230 may generate a diagnosis result based on the prediction results obtained using the diagnostic models. For example, the processor 230 may obtain prediction results for the presence of conjunctival redness, the presence of conjunctival edema, the presence of eyelid redness, the presence of eyelid edema, and the presence of caruncular edema using respective diagnostic models for diagnosing conjunctival redness, conjunctival edema, eyelid redness, eyelid edema, and caruncular edema for an eye image, and may generate a diagnosis result including a clinical activity score (CAS) for thyroid eye disease based on the obtained prediction results.

[0108] The processor 230 may transmit the presence and / or diagnosis of the disease to the user device 100 and / or an external device via the communication device 210.

[0109] The processor 230 may generate a diagnostic model using the training data and a model generation algorithm stored in the memory 220. Details regarding generating a diagnostic model using the training data are provided below.

[0110] The processor 230 may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a state machine, an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), and combinations thereof.

[0111] 4. Diagnostic Model 1 - Multi-label Model

[0112] Specific details of multiple label models for diagnosing diseases and / or conditions are provided below.

[0113] A multi-label model according to an embodiment may be a model in which multiple decision values ​​and one consensus value determined by multiple diagnosticians respectively are generated using multi-labeled training data for one diagnostic image.

[0114] A diagnostic image may refer to an image of a diagnostic subject. For example, a diagnostic image may be an image of all or part of a user's body acquired by a user using a user device. Without being limited thereto, a diagnostic image may refer to various types of images of a subject to be diagnosed, such as camera images, X-ray images, computed tomography (CT) images, and magnetic resonance imaging (MRI).

[0115] A diagnostic object may refer to an object to be analyzed to determine a user's status with respect to a disease and / or condition. For example, a diagnostic object may be various types of objects, such as the user's arms, legs, face, torso, and internal organs. A diagnostic object may be more specific and may be determined to be diverse and specific, such as the cornea, eyelids, and iris of the eye of the face, depending on the disease to be diagnosed.

[0116] A diagnostician may refer to a person who derives information about a disease and / or condition in a diagnostic image. For example, a diagnostician may be, but is not limited to, a medical staff member or a clinical manager. At least two diagnosticians, preferably three diagnosticians, may derive information from one diagnostic image.

[0117] The diagnostician may be determined from among those with a certain level of experience or more depending on the diagnosis target and / or diagnosis disease. For example, in the field of eye disease diagnosis, it has been found through research that the average diagnostic accuracy of ophthalmologists with more than 15 years of experience is about 86.24%, the average diagnostic accuracy of ophthalmology fellows with less than 5 years of experience is about 73.27%, and the average accuracy of ophthalmology residents is 56.04%. Therefore, it is preferable that the diagnostician who derives information on diseases and / or symptoms used to train the diagnostic model with eye disease as a diagnostic symptom is determined to be an ophthalmologist with more than 15 years of experience to ensure the accuracy of the diagnostic model.

[0118] The decision value may refer to information about the presence of a disease and / or condition as determined by an individual diagnostician who independently analyzes the diagnostic image. For example, when the diagnostic image is an eye image and the target disease is conjunctival redness, the decision value may be information about the presence of conjunctival redness.

[0119] The judgment value may be expressed as, but is not limited to, positive or negative for a disease and / or symptom. The judgment value may be expressed as the presence or absence of a disease, as true or false, as 1 or 0, or as one of the values ​​distinguished by multiple classes.

[0120] The call values ​​are determined by diagnosticians who independently analyze the diagnostic images, and as a result, the call values ​​may vary between diagnosticians.

[0121] On the other hand, the judgment value may be determined by the diagnostician meeting the actual user. Specifically, the diagnostician may determine the judgment value by checking the actual user with the naked eye. In this case, the diagnostic image may be obtained from the user after the diagnostician determines the judgment value, or may be obtained from the user before the diagnostician meets the actual user.

[0122] A consensus value may refer to information about the presence of a disease and / or condition that is determined by multiple diagnosticians analyzing diagnostic images together and reaching a consensus. For example, a consensus value may be a value determined by multiple diagnosticians discussing the same diagnostic images with each other at the same time and / or in the same space.

[0123] The multiple diagnosticians who determine the consensus value and the diagnostician who determines the decision value may refer to the same diagnostician, i.e., the multiple diagnosticians may analyze the diagnostic images independently of each other to determine the decision values, and then reach a mutual agreement to determine one consensus value.

[0124] The consensus value may be expressed in the same format as the decision value. For example, but not limited to, the consensus value may be expressed as positive or negative for a disease and / or condition. The consensus value may be expressed as the presence or absence of a disease, as true or false, as 1 or 0, or as one of the values ​​distinguished by multiple classes.

[0125] On the other hand, the consensus value may be determined by multiple diagnosticians meeting with the actual user. Specifically, the multiple diagnosticians may determine the consensus value by checking the actual user with the naked eye and discussing. In this case, the multiple diagnosticians may analyze the actual user independently of each other to determine their respective judgment values, and then discuss with each other to reach an agreement and determine the consensus value. In this case, the diagnostic image may be obtained from the user after the multiple diagnosticians have determined the consensus value, or may be obtained from the user before the multiple diagnosticians meet the actual user.

[0126] The label value refers to a value that is labeled with a diagnostic image in generating training data used to generate a diagnostic model, i.e., the training data may include diagnostic images as training input data and label values ​​as training result data.

[0127] The label value may be the judgment value and / or the consensus value itself. Without being limited thereto, the label value may be a value determined based on the judgment value and / or the consensus value. For example, the judgment value and / or the consensus value are values ​​indicating positive, and the label value may be a value of 1. When the judgment value and / or the consensus value are values ​​indicating negative, the label value may be a value of 0. The specific label value is not limited thereto, and may be determined in various manners according to the algorithm design of the diagnostic model.

[0128] Meanwhile, the person who determines the label value may be different from the diagnostician. For example, after the diagnostician derives the judgment value and the consensus value for the diagnostic image, an artificial intelligence engineer may determine the label value based on the judgment value and the consensus value of the diagnostician. As another example, when the diagnostician sends the judgment value and the consensus value to the server, the server may determine the label value based on the judgment value and the consensus value, and perform labeling on the diagnostic image.

[0129] Since the number of decision values ​​and agreement values ​​may be at least three, at least three label values ​​may be multi-labeled for the diagnostic images as training data used to generate the diagnostic model.

[0130] The training data used to generate the diagnostic model is described with reference to FIGS.

[0131] FIG. 4 illustrates a process for obtaining a decision score for each diagnostician, according to an embodiment.

[0132] 4, for the same diagnostic image 310, multiple diagnosticians 321, 322, and 323 independently diagnose the condition related to the disease and / or symptoms and determine respective decision values ​​331, 332, and 333. The decision values ​​331, 332, and 333 may be different values ​​since they are determined independently by the multiple diagnosticians 321, 322, and 323. For example, the first diagnostician 321 and the second diagnostician 322 may determine positive, and the third diagnostician 323 may determine negative.

[0133] While FIG. 4 shows three diagnosticians 321, 322 and 323, no limitation is imposed thereon and there may be two or four or more diagnosticians.

[0134] FIG. 5 illustrates a process for obtaining consensus values ​​from multiple diagnosticians, according to an embodiment.

[0135] Referring to FIG. 5, multiple diagnosticians 321 , 322 and 323 reach a mutual agreement on the status of diseases and / or symptoms in a diagnostic image 310 , thereby determining a single consensus value 340 .

[0136] In FIG. 5, multiple diagnosticians 321, 322, and 323 who diagnose diagnostic image 310 and determine consensus value 340 may be the same as multiple diagnosticians 321, 322, and 323 in FIG. 4 who independently diagnose diagnostic image 310 and determine decision values ​​331, 332, and 333.

[0137] The diagnostic image 310 diagnosed by multiple diagnosticians 321, 322 and 323 in FIG. 5 may be identical to the diagnostic image 310 used by the diagnosticians 321, 322 and 323 to determine the respective decision values ​​331, 332 and 333 in FIG.

[0138] That is, for the same diagnostic image 310, the diagnosticians 321, 322, and 323 may independently determine their respective decision values ​​331, 332, and 333 for the presence of a disease, and the same diagnosticians 321, 322, and 323 may determine the consensus value 340 by reaching a consensus for the presence of a disease for the same diagnostic image 310. For example, in determining the consensus value 340, the diagnosticians 321, 322, and 323 may reach a mutual agreement at the same time and / or in the same space. Specifically, the consensus value 340 may be determined by a majority vote of the diagnosticians 321, 322, and 323, or the consensus value 340 may be determined based on a minority judgment through discussion and agreement between the diagnosticians 321, 322, and 323. As another example, the consensus value 340 may be determined by a majority vote of the decision values ​​331, 332, and 333 of the diagnosticians 321, 322, and 323.

[0139] FIG. 6 is a diagram illustrating training data used to train a diagnostic model, according to an embodiment.

[0140] 6, the training data 350 used to generate the diagnostic model may include a diagnostic image 310, and a plurality of decision values ​​331, 332, and 333 and a consensus value 340 labeled on the diagnostic image 310. That is, the training data 350 may be generated by multi-labeling a plurality of data 360 including a plurality of decision values ​​331, 332, and 333 and a consensus value 340 on the diagnostic image 310. In this specification, multi-labeling may mean that the respective decision values ​​331, 332, and 333 and the consensus value 340 of a plurality of diagnosticians 321, 322, and 323 for the same symptom are associated with one diagnostic image 310. In this case, the diagnostic image 310 may be training input data, and the multi-labeled plurality of decision values ​​331, 332, and 333 and the consensus value 340 may be training result data.

[0141] FIG. 6 shows that the decision values ​​331, 332, and 333 and the consensus value 340 themselves are labeled for the diagnostic image 310, but this is not a limitation. The label values ​​corresponding to the decision values ​​331, 332, and 333 and the consensus value 340 may be labeled for the diagnostic image 310. For example, among the decision values ​​and the consensus value, a value of 1 may be determined as a label value for a value determined to have a disease, and a value of 0 may be determined as a label value for a value determined to not have a disease, and the label values ​​may be labeled for the diagnostic image 310. In this case, when the first decision value is a value determined to have a disease, the second decision value is a value determined to not have a disease, the third decision value is a value determined to not have a disease, and the consensus value is a value determined to have a disease, a value of [1, 0, 0, 1] may be labeled for the diagnostic image 310. However, the specific label values ​​and the labeling method are not limited thereto.

[0142] FIG. 6 shows that the same diagnostic image 310 as that used for diagnosis by multiple diagnosticians 321, 322 and 323 in FIG. 4 and 5 is used as training data. However, no limitation is imposed thereon. The diagnostic image 310 used as training data may be a diagnostic image that is subjected to separate pre-processing. For example, the diagnostic image used by multiple diagnosticians for disease diagnosis may be a whole face image, but when the disease is an eye-related disease, the diagnostic image used as training data may be an image obtained by cropping the whole face image to leave the eye area. That is, when the diagnostician determines the state of the disease, the diagnostic image itself may be used to facilitate the judgment. In the generation of the diagnostic model, a separate pre-processing may be performed on the diagnostic image to exclude unnecessary judgment elements in the diagnostic image, which increases the accuracy of the generated diagnostic model.

[0143] On the other hand, FIG. 6 shows that the first to third judgment values ​​331, 332, and 333 and the agreement value 340 for one type of disease and / or symptom are labeled for the diagnostic image 310, but this is not a limitation. The judgment values ​​and the agreement values ​​for at least two types of diseases and / or symptoms may be labeled for one diagnostic image. For example, the first to third judgment values ​​and the first agreement value for the first disease, and the fourth to sixth judgment values ​​and the second agreement value for the second disease may be labeled for one diagnostic image. As another example, the first to third judgment values ​​and the first agreement value for the first disease, the fourth to sixth judgment values ​​and the second agreement value for the second disease, and the seventh to ninth judgment values ​​and the third agreement value for the third disease may be labeled for one diagnostic image. The specific number of diseases is not limited to the above range.

[0144] FIG. 7 illustrates a process of predicting disease by using a trained diagnostic model, according to an embodiment.

[0145] Referring to FIG. 7, the diagnostic model 410 is Will beThe diagnostic model 410 may be configured to output a number of predictions 431, 432, 433, and 440 for the diagnostic image 420. In particular, the diagnostic model 410 may be configured to output a number of predictions 431, 432, 433, and 440 for the diagnostic image 420. Will be For a diagnostic image 420, the system may be configured to output predicted values ​​431, 432, and 433 corresponding to the respective decision values ​​of a plurality of diagnosticians, and a consensus predicted value 440 corresponding to the consensus value of the plurality of diagnosticians.

[0146] For example, the diagnostic model 410 may be

[0147] The model may be generated using algorithms such as a Convolutional Neural Network (CNN), a Vision Transformer, a Support Vector Machine (SVM), a Random Forest, a Gradient Boosting algorithm, an Artificial Neural Network (ANN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), ResNet, VGG, GoogLeNet, and / or MobileNet.

[0148] On the other hand, the diagnostic image 420 input to the diagnostic model 410 may be a diagnostic image that has been subjected to a separate pre-processing performed on the diagnostic image used as training data, i.e., before the diagnostic image is input to the diagnostic model 410, pre-processing may be performed on the diagnostic image, and the pre-processed diagnostic image may be input to the diagnostic model 410.

[0149] 7 shows that because there are three diagnosticians, there are three predictive values ​​431, 432, and 433 corresponding to the diagnosis values ​​of each of the multiple diagnosticians. If there are two or four or more diagnosticians, there may be two or four or more predictive values ​​corresponding to the diagnosis values.

[0150] The diagnostic model 410 may be generated using the training data 350 described with reference to FIG. 6. In the training process, different weights for the loss values ​​may be set for each output node. Specifically, in the training of the diagnostic model 410, the same loss weights may be set for all the multiple output nodes. However, a high loss weight may be set for the output node corresponding to the consensus predicted value. For example, as shown in FIG. 7, when there are four output nodes, the loss weight for each of the output nodes may be set to 0.25. Alternatively, the loss weight for the node corresponding to the consensus predicted value 440 may be set to 0.7, and each of the loss weights for the nodes corresponding to the other predicted values ​​431, 432, and 433 may be set to 0.1. Alternatively, the loss weight for the node corresponding to the consensus predicted value 440 may be set to 0.5, and each of the loss weights for the nodes corresponding to the other predicted values ​​431, 432, and 433 may be set to 0.5 / 3. The specific values ​​for the loss weights are not limited to the values ​​described above, and the specific values ​​for the loss weights may be determined by appropriately adjusting the values ​​through experiments.

[0151] On the other hand, Fig. 7 shows that the first to third predicted values ​​431, 432, and 433 and the consensus predicted value 440 for one type of disease and / or symptom are output from the diagnostic model 410, but this is not a limitation. The predicted value and the consensus predicted value for each of at least two types of diseases and / or symptoms may be output from one diagnostic model. For example, when a diagnostic model is generated using a diagnostic image in which the first to third judgment values ​​and the first consensus value for a first disease and the fourth to sixth judgment values ​​and the second consensus value for a second disease are multiply labeled, the diagnostic model may output the predicted value and the consensus predicted value for each of the two types of diseases. As another example, when a diagnostic model is generated using a diagnostic image that is multiply labeled with the first to third judgment values ​​and the first consensus value for a first disease, the fourth to sixth judgment values ​​and the second consensus value for a second disease, and the seventh to ninth judgment values ​​and the third consensus value for a third disease, the diagnostic model may output a predicted value and a consensus predicted value for each of three types of diseases. The specific number of diseases is not limited to the above range.

[0152] A final predicted value for determining the presence of a disease can be determined based on the predicted values ​​431, 432, and 433 output by the diagnostic model 410 and the consensus predicted value 440. For example, among the output values, the consensus predicted value 440 can be determined as the final predicted value. Since one consensus predicted value 440 corresponding to the consensus value determined by multiple diagnosticians reaching a consensus is used as the final predicted value, the final predicted value can be obtained without additional processing of the output value of the diagnostic model 410, thereby reducing the accuracy loss caused by the additional processing.

[0153] In addition, the diagnostic model 410 according to the embodiment may learn more information than a model trained using only one decision value or one consensus value of each diagnostician, because each decision value and consensus value of multiple diagnosticians are learned in one model. Specifically, since the decision values ​​of multiple diagnosticians may affect the consensus value in the learning process, the diagnostic model may learn more information, such as definite positive, ambiguous positive, ambiguous negative, and definite negative, rather than simply distinguishing positive and negative. That is, the process of determining each decision value by multiple diagnosticians may be applied to the final prediction value, and as a result, the prediction performance may be improved.

[0154] In addition, since the respective judgment values ​​of multiple diagnosticians are learned in one model and the respective judgment values ​​may affect each other during the learning process, the diagnostic model 410 according to the embodiment can compensate for errors that may be caused by some of the multiple diagnosticians during the process of determining the judgment value, thereby increasing the accuracy of the predicted value corresponding to the judgment value.

[0155] Meanwhile, a final prediction value for determining the presence of a disease and / or a symptom may be determined based on a value obtained by applying weights to the prediction values ​​431, 432, and 433 output by the diagnostic model 410 and the consensus prediction value 440, respectively, and adding the values. For example, when the sum of the prediction values ​​431, 432, and 433 and the consensus prediction value 440 is equal to or greater than a threshold, it may be determined as positive. When the sum is less than the threshold, it may be determined as negative. Meanwhile, in setting the weights, the weight for the consensus prediction value 440 may be set higher than the weights for the other prediction values ​​431, 432, and 433, and the values ​​may be added.

[0156] Meanwhile, the final predicted value for determining the presence of a disease may be determined by a majority vote of the predicted values ​​431, 432, and 433 and the consensus predicted value 440 output by the diagnostic model 410. For example, when all the majority votes of the predicted values ​​and the consensus predicted value are positive values, the final predicted value may be determined as positive, or when the majority vote is negative, the final predicted value may be determined as negative. Meanwhile, when the number of positive values ​​is equal to the number of negative values, the final predicted value may be determined according to the consensus predicted value.

[0157] Meanwhile, the final predicted value for determining the presence of a disease may be determined based on the average value of the predicted values ​​431, 432, and 433 output by the diagnostic model 410 and the consensus predicted value 440. Without being limited thereto, the final predicted value for determining the presence of a disease may be determined based on other statistics obtained using the predicted values ​​431, 432, and 433 output by the diagnostic model 410 and the consensus predicted value 440.

[0158] Depending on the design purpose and design method, there may be various diagnostic models for predicting diseases and / or symptoms based on diagnostic images.

[0159] The present inventors have experimented the performance of the multi-label model according to the present disclosure by using a symptom diagnosis model for eye diseases. Specifically, the diagnostic image is a user's eye image, the diagnostic target is the eyelid, and the judgment value is set to the presence of eyelid redness.

[0160] Three ophthalmologists were set as diagnosticians, and comparative experiments were performed to experiment the performance of a first doctor model trained only with the judgment values ​​of the first doctor, a second doctor model trained only with the judgment values ​​of the second doctor, a third doctor model trained only with the judgment values ​​of the third doctor, and a consensus value model trained only with the consensus values ​​of the first to third doctors.

[0161] Each model was generated using training data including 1020 diagnostic images and the ResNet algorithm. Specifically, the training data was randomly divided in a ratio of training:validation:test=7:1:2, and the experiment was performed with 30 iterations. That is, in one experiment, each model was generated using training data including 714 training data, 102 validation data, and 204 test data.

[0162] The F1 score, accuracy, sensitivity, specificity, PPV, and NPV of the compared models are shown in Table 1 below.

[0163] [Table 1] [Table 1]

[0164] Meanwhile, the performance of multi-label models generated according to embodiments of the present disclosure was experimented.

[0165] The multi-label model was generated using the ResNet algorithm and training data including 1020 diagnostic images with multiple labels of the judgment values ​​of the first to third doctors and the consensus values ​​of the first to third doctors. Specifically, the training data was randomly divided in a ratio of training:validation:test=7:1:2, and an experiment was performed with 30 iterations. That is, in one experiment, the multi-label model was generated using training data including 714 training data, 102 validation data, and 204 test data. The diagnostic images were the same as the diagnostic images used to generate each of the above models.

[0166] The F1 score, accuracy, sensitivity, specificity, PPV, and NPV of the first to third prediction values ​​and the consensus prediction value, which are the output values ​​of the multi-label model generated according to an embodiment of the present disclosure, are shown in Table 2 below.

[0167] [Table 2] [Table 2]

[0168] Referring to [Table 1] and [Table 2], it can be seen that the performance of the diagnostic model generated using training data in which the judgment values ​​and consensus values ​​of the first to third diagnosticians are multi-labeled for the diagnostic image is better than that of the compared models. In addition, it can be seen that the best performance is achieved when the final prediction value is determined based on the consensus prediction value output by the multi-label model. On the other hand, even if the target disease of the experimental eye disease diagnostic model is changed to a different disease, it can be predicted that a similar trend will be shown.

[0169] On the other hand, referring to Figures 6 and 7, it has been described that the diagnostic model is generated using training data in which the judgment values ​​and consensus values ​​of multiple diagnosticians are multi-labeled for the diagnostic image. However, as shown in Figures 8 and 9, the diagnostic model can be generated using training data in which only the judgment values ​​of multiple diagnosticians, excluding the consensus value, are multi-labeled for the diagnostic image. Even in this case, the significant improvement as described above can still be realized.

[0170] Specifically, FIG. 8 is a diagram illustrating training data used to train a diagnostic model according to an embodiment.

[0171] 8, the training data 550 used to generate the diagnostic model may include a diagnostic image 510 and a plurality of judgment values ​​521, 522, and 523 labeled for the diagnostic image 510. That is, the training data 550 may be generated by multi-labeling a plurality of data 560 for the diagnostic image 510. In this specification, multi-labeling may mean that the respective judgment values ​​331, 332, and 333 of a plurality of diagnosticians 321, 322, and 323 for the same symptom are associated with one diagnostic image 510. In this case, the diagnostic image 510 may be training input data, and the multi-labeled multiple judgment values ​​521, 522, and 523 may be training result data. The diagnostic image and the multiple judgment values ​​have been described above, so redundant description will be omitted.

[0172] 8 shows that the first to third judgment values ​​521, 522, and 523 for one type of disease and / or symptom are labeled for the diagnostic image 510, but is not limited thereto. Judgment values ​​for at least two types of disease and / or symptom may be labeled for one diagnostic image, and the details are similar to those described with reference to FIG. 6, so redundant description will be omitted.

[0173] FIG. 9 illustrates a process of predicting disease by using a trained diagnostic model, according to an embodiment.

[0174] 9, the diagnostic model 610 may be configured to output a plurality of predicted values ​​631, 632, and 633 for an input diagnostic image 620. Specifically, the diagnostic model 610 may be configured to output predicted values ​​631, 632, and 633 corresponding to the respective judgment values ​​of a plurality of diagnosticians for the input diagnostic image 620. The diagnostic model, the diagnostic image, and the predicted values ​​have been described above, and therefore redundant description will be omitted.

[0175] 9 shows that the first to third predicted values ​​631, 632, and 633 for one type of disease and / or symptom are output from the diagnostic model 610, but is not limited thereto. Predicted values ​​for each of at least two types of diseases and / or symptoms may be output from one diagnostic model, and the details thereof are similar to those described with reference to FIG. 7, so redundant description will be omitted.

[0176] The present inventors have experimented the performance of a multi-label model using only diagnostician's judgment value according to the present disclosure by using a symptom diagnosis model for eye diseases. Specifically, the diagnostic image is a user's eye image, the diagnostic target is the caruncle, and the judgment value is set to the presence of caruncle edema.

[0177] Three ophthalmologists were set as diagnosticians, and comparative experiments were performed to compare the performance of a first doctor model trained only with the judgment values ​​of the first doctor, a second doctor model trained only with the judgment values ​​of the second doctor, and a third doctor model trained only with the judgment values ​​of the third doctor, as well as the values ​​and consensus values ​​determined by majority vote of the output values ​​of the first to third doctor models.

[0178] Each model was generated using training data including 1020 diagnostic images and the ResNet algorithm. Specifically, the training data was randomly divided in a ratio of training:validation:test=7:1:2, and the experiment was performed with 30 iterations. That is, in one experiment, each model was generated using training data including 714 training data, 102 validation data, and 204 test data.

[0179] The F1 scores, accuracy, sensitivity, specificity, PPV, and NPV of the compared experiments are shown in Table 3 below.

[0180] [Table 3] [Table 3]

[0181] Meanwhile, the performance of the multi-label model was tested using only the judgment values ​​of three doctors according to an embodiment of the present disclosure.

[0182] The multi-label model was generated using the ResNet algorithm and training data including 1020 diagnostic images with multiple labels of the first to third doctor's judgment values. Specifically, the training data was randomly divided in a ratio of training:validation:test=7:1:2, and an experiment was performed with 30 iterations. That is, in one experiment, the multi-label model was generated using training data including 714 training data, 102 validation data, and 204 test data. The diagnostic images were the same as the diagnostic images used to generate each of the above models.

[0183] The results of an experiment using the performance of the first to third predicted values, which are the output values ​​of a multi-label model generated according to an embodiment of the present disclosure, and a comparison between the value determined by majority vote of the first to third predicted values ​​and the consensus value, are shown in Table 4 below.

[0184] [Table 4] [Table 4]

[0185] Referring to [Table 3] and [Table 4], it can be seen that the performance of the diagnostic model generated using the training data in which the judgment values ​​of multiple diagnosticians are multi-labeled was, on average, better than that of the compared models. That is, it can be seen that significant improvement can be achieved by generating a diagnostic model using training data in which the judgment values ​​of multiple diagnosticians are multi-labeled for diagnostic images.

[0186] 5. Diagnostic Model 2 - Label Facilitation Model

[0187] Specific details of the label facilitation model for diagnosing diseases and / or conditions are described below.

[0188] A label smoothing model according to an embodiment may be a model that uses label values ​​for consensus values ​​determined by multiple diagnosticians for one diagnostic image, and is generated using training data in which smoothing is performed on the label values ​​taking into account multiple decision values ​​respectively determined by multiple diagnosticians, and the smoothed label values ​​are labeled for the diagnostic image.

[0189] The diagnostic image, diagnostic subject, diagnostician, decision value, agreement value, and label value have been described above in the section describing the multi-label model, so redundant description will be omitted.

[0190] The training data used to generate the diagnostic model is described with reference to FIG.

[0191] FIG. 10 is a diagram illustrating training data used to train a diagnostic model, according to an embodiment.

[0192] 10, first, multiple diagnosticians may independently diagnose the disease and / or symptom status for the same diagnostic image 710 and determine decision values ​​731, 732, and 733 for each diagnostician. In addition, the multiple diagnosticians may determine one consensus value 720 by reaching a mutual agreement on the disease and / or symptom status for the same diagnostic image 710.

[0193] The diagnosticians who determine the multiple decision values ​​731, 732, and 733 may be the same as the multiple diagnosticians who determine the consensus value 720, the diagnostic image 710 used to determine the multiple decision values ​​731, 732, and 733 may be the same as the diagnostic image 710 used to determine the consensus value 720, and the diagnostic image 710 used as training data may be a diagnostic image that has been subjected to separate pre-processing. This has been described above, so redundant description will be omitted.

[0194] 10, the label value labeled for the diagnostic image 710 may be a smoothed consensus value 740 determined based on a plurality of decision values ​​731, 732, and 733 and a consensus value 720. For example, when the consensus value 720 is positive and the decision values ​​731, 732, and 733 determined by a plurality of diagnosticians independently are unanimously positive, the smoothed consensus value 740 may be determined to a value of 1. When the consensus value 720 is negative and the decision values ​​731, 732, and 733 determined by a plurality of diagnosticians independently are unanimously negative, the smoothed consensus value 740 may be determined to a value of 0. When the consensus value 720 is positive and the plurality of decision values ​​731, 732, and 733 are not unanimous, the smoothed consensus value 740 may be determined to a value of 0.8. When the consensus value 720 is negative and the multiple decision values ​​731, 732, and 733 are not unanimous, the smoothed consensus value 740 may be determined to a value of 0.2. Without limitation, when the consensus value 720 is positive and the multiple decision values ​​731, 732, and 733 are not unanimous, the smoothed consensus value 740 may be determined to a value of 0.7. On the other hand, when the consensus value 720 is negative and the multiple decision values ​​731, 732, and 733 are not unanimous, the smoothed consensus value 740 may be determined to a value of 0.3. 731, 732 and 733 Considering the distribution of 720 can be determined by appropriately adjusting, but is not limited to, the values ​​described above.

[0195] On the other hand, Figure 10 shows three judgment values. 731, 732 and 733 , i.e., three diagnosticians are shown, but no limitation is imposed and there may be two or four or more diagnosticians. In this case, Judgment value The number can be 2 or 4 or more.

[0196] On the other hand, Fig. 10 shows that the agreement value 720 and the smoothed agreement value 740 determined based on the first to third judgment values ​​731, 732 and 733 for one type of disease and / or symptom are labeled for the diagnostic image 710, but this is not limited thereto. At least two smoothed agreement values ​​determined based on the judgment values ​​and agreement values ​​for at least two types of diseases and / or symptoms may be labeled for one diagnostic image. For example, a first smoothed agreement value determined based on the first agreement value and the first to third judgment values ​​for a first disease, and a second smoothed agreement value determined based on the second agreement value and the fourth to sixth judgment values ​​for a second disease may be labeled for one diagnostic image. As another example, the first smoothed consensus value determined based on the first consensus value and the first to third judgment values ​​for the first disease, the second smoothed consensus value determined based on the second consensus value and the fourth to sixth judgment values ​​for the second disease, and the third smoothed consensus value determined based on the third consensus value and the seventh to ninth judgment values ​​for the third disease may be labeled multiple times for one diagnostic image. The specific number of diseases is not limited to the above range.

[0197] FIG. 11 illustrates a process of predicting disease by using a trained diagnostic model, according to an embodiment.

[0198] Referring to FIG. 11, the diagnostic model 810 is Will be The diagnostic image 820 may be configured to output a predicted value 830 corresponding to the smoothed consensus value for the diagnostic image 820 .

[0199] For example, the diagnostic model 810 may be a model generated using an algorithm such as a convolutional neural network (CNN), a vision transformer, a support vector machine (SVM), a random forest, a gradient boosting algorithm, an artificial neural network (ANN), a deep neural network (DNN), a recurrent neural network (RNN), ResNet, VGG, GoogLeNet, and / or MobileNet.

[0200] On the other hand, as explained above, the diagnostic image 820 input to the diagnostic model 810 may be a diagnostic image that has been subjected to separate pre-processing performed on the diagnostic image used as training data, so redundant explanation will be omitted.

[0201] 11 shows that a predicted value 830 for one type of disease and / or symptom is output from the diagnostic model 810, but this is not a limitation. A predicted value for each of at least two types of diseases and / or symptoms may be output from one diagnostic model. For example, when a diagnostic model is generated using a diagnostic image in which a first smoothed consensus value determined based on a first consensus value and the first to third judgment values ​​for a first disease, and a second smoothed consensus value determined based on a second consensus value and the fourth to sixth judgment values ​​for a second disease are multiply labeled, the diagnostic model may output a predicted value for each of two types of diseases. As another example, when a diagnostic model is generated using a diagnostic image in which a first smoothed consensus value determined based on the first consensus value and the first to third judgment values ​​for a first disease, a second smoothed consensus value determined based on the second consensus value and the fourth to sixth judgment values ​​for a second disease, and a third smoothed consensus value determined based on the third consensus value and the seventh to ninth judgment values ​​for a third disease are multiply labeled, the diagnostic model may output a predicted value for each of three types of diseases. The specific number of diseases is not limited to the above range.

[0202] A final predicted value for determining the presence of a disease can be determined based on the predicted value 830 output by the diagnostic model 810. Since the predicted value 830 corresponding to the smoothed consensus value determined by considering the judgment values ​​and consensus values ​​of multiple diagnosticians is used as the final predicted value, the final predicted value can be obtained without additional processing of the output value of the diagnostic model 810, thereby reducing the accuracy loss caused by the additional processing.

[0203] In addition, the predicted value 830 obtained from the diagnostic model 810 is a predicted value corresponding to a smoothed consensus value to which the judgment values ​​of multiple diagnosticians are applied, so that the final predicted value may reflect the judgment values ​​of multiple diagnosticians.

[0204] Depending on the design purpose and design method, there may be various diagnostic models for predicting diseases and / or symptoms based on diagnostic images.

[0205] The present inventors have experimented with the performance of the label smoothing model according to the present disclosure by using a symptom diagnosis model for eye diseases. Specifically, the diagnostic image is a user's eye image, the diagnostic target is the eyelid, and the judgment value is set to the presence of eyelid redness.

[0206] Three ophthalmologists were set as diagnosticians, and the performance of a first doctor model trained only using the judgment values ​​of the first doctor, a second doctor model trained only using the judgment values ​​of the second doctor, a third doctor model trained only using the judgment values ​​of the third doctor, and a consensus value model trained only using the consensus values ​​of the first to third doctors were tested through comparative experiments.

[0207] Each model was generated using training data including 1020 diagnostic images and the ResNet algorithm. Specifically, the training data was randomly divided in a ratio of training:validation:test=7:1:2, and the experiment was performed with 30 iterations. That is, in one experiment, each model was generated using training data including 714 training data, 102 validation data, and 204 test data.

[0208] The F1 score, accuracy, sensitivity, specificity, PPV, and NPV of the compared models are shown in Table 5 below, which is the same experimental result as Table 1.

[0209] [Table 5] [Table 5]

[0210] Meanwhile, the performance of the label smoothing model according to the embodiment of the present disclosure was experimented.

[0211] The label smoothing model uses a smoothed consensus value of 1, 0.8, 0.2, or 0 based on the consensus value determined by the first to third doctors reaching an agreement and the judgment value determined by the first to third doctors, respectively. Labeling The multi-label model was generated using training data including 1020 diagnostic images and the ResNet algorithm. Specifically, the training data was randomly divided in a ratio of training:validation:test = 7:1:2, and the experiment was performed with 30 iterations. That is, in one experiment, the multi-label model was generated using training data including 714 training data, 102 validation data, and 204 test data. The diagnostic images were the same as those used to generate each of the above models.

[0212] The F1 score, accuracy, sensitivity, specificity, PPV, and NPV of the predictions output by the label smoothing model generated according to an embodiment of the present disclosure are shown in Table 6 below.

[0213] [Table 6] [Table 6]

[0214] Referring to Tables 5 and 6, it can be seen that the performance of the label smoothing model is better than that of the compared models. On the other hand, even if the target disease of the experimental eye disease diagnosis model is changed to a different disease, it can be expected that a similar trend will be shown.

[0215] 6. System for predicting clinical activity score (CAS) for thyroid eye disease

[0216] Below, a system for predicting a clinical activity score for thyroid eye disease by using the diagnostic model described above is described in detail.

[0217] The clinical activity score of thyroid eye disease can be determined by considering a total of seven items. Specifically, the seven items include conjunctival redness, conjunctival swelling, lacrimal caruncle swelling, eyelid redness, eyelid swelling, spontaneous retrobulbar pain, and pain when attempting to gaze upward or downward. Each symptom is assigned a score of 1 for positive or 0 for negative, and all scores are added together to determine the final clinical activity score.

[0218] FIG. 12 is a diagram illustrating a system 20 for predicting clinical activity scores for thyroid eye disease, according to an embodiment.

[0219] The system 20 may obtain information from the user about spontaneous retrobulbar pain and pain when attempting to gaze upward or downward, among a total of seven items considered in determining a clinical activity score for thyroid eye disease. Specifically, although not shown in Fig. 12, the system 20 may display a graphical user interface (GUI) for receiving user input on the display of the user device, and when the user inputs information into the user device, the system 20 may obtain information about each symptom from the user device.

[0220] The system 20 may use independent diagnostic models to perform predictions for conjunctival redness, swelling of conjunctiva, swelling of lacrimal caruncle, redness of eyelid, and swelling of eyelid in the clinical activity score for thyroid eye disease. Specifically, a facial image representing the user's eyes obtained from a user device and five diagnostic models for predicting each symptom may be used to predict a score for each of the five symptoms.

[0221] To predict the score for each symptom, the system 20 may first obtain a facial image and / or an eye image representing the user's eyes. An eye image may refer to an image representing the conjunctiva, white of the eye, cornea, iris, eyelid, eyebrow, eyelash, exposed eyeball, eye contour, exposed cornea, and exposed conjunctiva. Alternatively, an eye image may refer to an image of the user's upper eyelid, lower eyelid, and eyeball exposed by the upper and lower eyelids (exposed eyeball).

[0222] A user uses a user device to directly photograph his or her face to obtain a facial image, which is then transmitted to the system 20, which obtains an eye image.

[0223] For example, a user may acquire a facial image of himself / herself using a user device, the user device may perform pre-processing on the facial image to acquire an eye image, the user device may transmit the eye image to the system 20, and the system 20 may acquire the eye image. As another example, a user may acquire a facial image of himself / herself using a user device, the user device may transmit the facial image to the system 20, and the system 20 may perform pre-processing on the facial image to acquire an eye image. Without limitation, a user may acquire an eye image of himself / herself directly using a user device, and the user device may transmit the eye image to the system 20.

[0224] The user device may provide a shooting guide to enable the user to easily capture a face image. The shooting guide may be displayed on the display of the user device or may be provided to the user by voice and / or sound. For example, the user device may display a face outline and / or a desired eye position on a preview image captured by photographing the user, thereby guiding the user to easily and intuitively adjust the shooting angle, position, and / or direction of the user device.

[0225] The system 20 may obtain a prediction value for each of the five types of symptoms using the first to fifth diagnostic models. Specifically, the first to fifth diagnostic models may be a conjunctival redness diagnostic model, a conjunctival edema diagnostic model, a caruncular edema diagnostic model, an eyelid edema diagnostic model, and an eyelid redness diagnostic model, respectively.

[0226] Before inputting the acquired facial image into the first to fifth diagnostic models, the system 20 may perform two different types of pre-processing on the facial image. Specific details of the pre-processing will be described with reference to FIGS. 13 and 14.

[0227] 13 and 14 are diagrams illustrating an image pre-processing process, according to an embodiment.

[0228] The system 20 may perform a first pre-processing on the image input to the model to diagnose conjunctival redness, chemosis, and caruncular edema, where the eyeball region is the main target for judgment. With reference to Fig. 13, the first pre-processing may mean a pre-processing for masking the pupil and skin, and the pre-processed image may be an eyeball region image.

[0229] The system 20 may perform a second pre-processing on the image input to the model for diagnosing eyelid edema and eyelid redness, where the eyelid region is the main target for judgment. With reference to Fig. 14, the second pre-processing may mean a pre-processing for masking the eyeball region, and the pre-processed image may be an eyelid region image.

[0230] The system 20 may input each preprocessed image into the first to fifth diagnostic models to obtain a predicted value for the first to fifth symptoms. The system 20 may obtain scores for conjunctival redness, conjunctival edema, caruncular edema, eyelid redness, and eyelid edema based on the predicted values ​​for the first to fifth symptoms, and may add up the obtained scores. The system 20 may further obtain a clinical activity score for thyroid eye disease by adding up scores obtained from the user for spontaneous retrobulbar pain and pain when attempting to gaze upward or downward.

[0231] The system 20 may transmit the acquired clinical activity score and / or information about thyroid eye disease to the user device and / or other external devices. For example, when the acquired clinical activity score is equal to or greater than the reference value, the system 20 may transmit a message to the user device indicating that the user has a risk of thyroid eye disease. Additionally, the system 20 may transmit a message to the user device to guide the user to visit a hospital due to the risk of thyroid eye disease. The user device may provide the message received from the system 20 to the user through a user interface.

[0232] The first to fifth diagnostic models used by the system 20 are models for diagnosing symptoms using images, and the first to fifth diagnostic models may be realized as the multi-label model and / or label smoothing model described above. Without being limited thereto, the first to fifth diagnostic models may be realized as various types of models for outputting predicted values ​​based on images. In addition, the first to fifth diagnostic models may not all be generated by the same algorithm. That is, at least some of the first to fifth diagnoses may be models generated by different algorithms, or all of the models may be models generated by different algorithms.

[0233] On the other hand, how to determine the training data and / or evaluation data to use the diagnostic model for actual diagnosis is an element that needs to be considered for safe use of the diagnostic model. Specifically, the performance of the diagnostic model based on the evaluation data needs to be equal to or higher than the standard, so that the diagnostic model is used for actual diagnosis. Therefore, the judgment of the evaluation data may require a high standard.

[0234] Therefore, the judgment value and / or consensus value included in the evaluation data for evaluating the performance of the diagnostic model may be a judgment value and / or consensus value determined by a diagnostician actually meeting a patient and diagnosing. The diagnostic image included in the evaluation data may be obtained from a patient after the diagnostician actually meets the patient and diagnoses, or before the diagnostician meets the patient. That is, the judgment value and / or consensus value labeled for the diagnostic image, which is the evaluation data, may be a value determined by multiple diagnosticians actually meeting a patient and diagnosing, rather than a value determined through the diagnostic image. Details regarding a diagnostician meeting a user in person to diagnose the user and determine the judgment value and consensus value have been described above, so redundant description will be omitted.

[0235] The decision values ​​and / or consensus values ​​included in the training data used to train the diagnostic model may also be decision values ​​and / or consensus values ​​determined by a diagnostician actually meeting and diagnosing a patient, as described above.

[0236] On the other hand, the training data may not necessarily require a high standard as the evaluation data. Therefore, the training data may include a judgment value and / or a consensus value determined by a diagnostician diagnosing a patient's image rather than an actual patient. Alternatively, the judgment value included in the training data may be a value determined by a diagnostician diagnosing a patient through a patient's image, and the consensus value may be a value determined by a majority vote of the judgment values.

[0237] The present disclosure described above is not limited to the embodiments and accompanying drawings, since various substitutions, modifications, and changes from the spirit of the present disclosure defined in the following claims by those skilled in the art are also included in the scope of the present disclosure. In addition, the embodiments described in this specification may not be applied in a limited manner, and all or part of the embodiments may be selectively combined, resulting in various modifications. Furthermore, the steps constituting each embodiment may be used individually or in combination with the steps constituting other embodiments. (Item 1) obtaining a face image including an eye region; obtaining a plurality of outcome values ​​using the facial image and a diagnostic model for predicting an eye-related condition, wherein the plurality of outcome values ​​includes a plurality of predicted values ​​and a single consensus predicted value; and determining the presence of the ocular-related condition based on the consensus predicted value of the plurality of outcome values; Equipped with wherein the diagnostic model is generated using training data in which a single diagnostic image is multi-labeled, the multi-labeled diagnostic model includes a plurality of judgment values ​​independently determined by two or more different diagnosticians regarding the presence of the eye-related condition in the same diagnostic subject, and a single consensus value determined by the diagnosticians through mutual agreement regarding the presence of the eye-related condition in the same diagnostic subject; wherein the plurality of predicted values ​​correspond to the plurality of decision values, and the consensus predicted value corresponds to the consensus value; wherein the eye-related symptom is at least one of conjunctival redness, conjunctival edema, caruncular edema, eyelid redness, and eyelid edema; Symptom prediction methods. (Item 2) 2. The method of claim 1, wherein at least a portion of the training data has the agreement value corresponding to a minority of the decision values ​​of the plurality of decision values. (Item 3) 2. The method of claim 1, wherein when training the diagnostic model, a loss weight for a node at which the consensus prediction value is output is set higher than a loss weight for each node at which the prediction value is output. (Item 4) The step of obtaining the plurality of result values ​​comprises: performing pre-processing on the face image for the eye region; and obtaining the plurality of result values ​​using the pre-processed image and the diagnostic model; The method according to item 1, comprising: (Item 5) The preprocessing steps are: performing pre-processing on the face image to mask pupils and skin when the eye-related condition is at least one of the conjunctival redness, the conjunctival edema, and the caruncular edema; and performing pre-processing on the face image to mask an eye region when the eye-related condition is at least one of the eyelid redness and the eyelid edema. The method according to item 4, comprising: (Item 6) The diagnosticians were three ophthalmologists with over 15 years of experience; The judgment value includes three values ​​corresponding to each diagnostician; The method according to item 1. (Item 7) 2. The method of claim 1, wherein at least one of the judgment value and the agreement value is a value determined by the diagnostician through an actual meeting with the diagnostic subject regarding the presence of the eye-related condition. (Item 8) 2. The method of claim 1, wherein at least one of the judgment value and the agreement value is a value determined by the diagnostician through a facial image of the diagnostic subject regarding the presence of the eye-related condition. (Item 9) 2. The method of claim 1, wherein the consensus value is determined after the diagnostician determines each of the call values. (Item 10) The method according to item 1, wherein the diagnostic model is generated using at least one algorithm of a convolutional neural network (CNN), a vision transformer, a support vector machine (SVM), a random forest, a gradient boosting algorithm, an artificial neural network (ANN), a deep neural network (DNN), a recurrent neural network (RNN), ResNet, VGG, GoogLeNet, and MobileNet.

Claims

1. A step of obtaining three or more individual judgment values ​​corresponding to independent judgments regarding the presence of eye-related symptoms in a subject and a consensus value corresponding to mutual agreement regarding the presence of the eye-related symptoms in the subject, wherein the independent judgments are made by three or more different ophthalmologists, the mutual agreement is reached by the three or more different ophthalmologists in the same physical area or the same virtual meeting, the independent judgments of the three or more different ophthalmologists include a majority of first judgments among the three or more different ophthalmologists and a minority of second judgments among the three or more different ophthalmologists, and the consensus value corresponds to the value of the minority of second judgments; A step of training a single diagnostic model using training data to output predictions of three or more individual judgment values ​​and one consensus value in response to an input image, wherein the training data includes clinical images multiple-labeled with the three or more individual judgment values ​​and the one consensus value; The step of obtaining an image representing at least one eye; A step of obtaining three or more individual predictive values ​​and one agreed predictive value based on the aforementioned images and the trained single diagnostic model; and A step of determining the presence of eye-related symptoms in the eye based on the aforementioned agreed-upon predicted values, Here, the clinical image represents at least one eye of a person, Here, the number of individual judgment values ​​is equal to the number of different ophthalmologists. Here, the aforementioned eye-related symptoms are one of the following: conjunctival redness, conjunctival edema, lacrimal caruncle edema, eyelid redness, or eyelid edema. A method for predicting eye-related symptoms that includes the following features.

2. The aforementioned training data includes multiple clinical images, A method for predicting eye-related symptoms according to claim 1.

3. The eye-related symptom prediction method according to claim 1, wherein when training the single diagnostic model, the loss weight for the node where the prediction of the agreed value is output is set higher than the loss weight for each node where the prediction of the individual judgment value is output.

4. The steps for obtaining the three or more individual forecast values ​​and the one agreed forecast value are: A step of performing preprocessing on the image of the eye region; and The step of obtaining the three or more individual predictive values ​​and the one agreed predictive value using the pre-processed images and the single diagnostic model. A method for predicting eye-related symptoms according to claim 1, including the method described in claim 1.

5. The steps for performing preprocessing are: When the eye-related symptom is one of the conjunctival redness, conjunctival edema, or lacrimal caruncle edema, the step of preprocessing the image to mask the pupil and skin; and If the eye-related symptom is one of the eyelid redness or eyelid edema, the step is to perform preprocessing on the image to mask the eyeball region. A method for predicting eye-related symptoms according to claim 4, including the method described in claim 4.

6. Each of the three or more different ophthalmologists has more than 15 years of experience, The number of individual predicted values ​​is three, and each of the three individual predicted values ​​corresponds to three different ophthalmologists. A method for predicting eye-related symptoms according to claim 1.

7. The method for predicting eye-related symptoms according to claim 1, wherein at least one of the independent judgments is made by the ophthalmologist actually meeting with the person.

8. The method for predicting eye-related symptoms according to claim 1, wherein at least one of the independent judgments is made by the ophthalmologist based on the clinical images of the person.

9. The method for predicting eye-related symptoms according to claim 1, wherein the mutual agreement is made by the ophthalmologist after the ophthalmologist has made the independent judgment.

10. The method for predicting eye-related symptoms according to claim 1, wherein the single diagnostic model is generated using the ResNet algorithm.

11. A memory configured to store images representing at least one eye and a single diagnostic model; and Based on the aforementioned image and the single diagnostic model, obtain three or more individual predicted values ​​and one agreed predicted value, and Based on the aforementioned agreed-upon predicted values, the presence of eye-related symptoms in the eye is determined. Processors configured in such a way Equipped with, Here, the single diagnostic model is trained using training data to output predictions of three or more individual decision values ​​and one consensus value in response to an input image. Here, the training data includes clinical images that are multiplely labeled with three or more individual judgment values ​​and one consensus value. Here, the clinical image represents at least one eye of a person, Here, each of the three or more individual judgment values ​​corresponds to the independent judgment of three or more different ophthalmologists regarding the presence of the eye-related symptoms in the same person. Here, the number of individual judgment values ​​is equal to the number of different ophthalmologists. Here, the one agreed value corresponds to the mutual agreement of the three or more different ophthalmologists present in the same physical area or the same virtual meeting regarding the presence of the eye-related symptoms in the same person, the independent judgments of the three or more different ophthalmologists include a majority of first judgments among the three or more different ophthalmologists and a minority of second judgments among the three or more different ophthalmologists, the one agreed value corresponds to the value of the minority of second judgments, and Here, the aforementioned eye-related symptoms are one of the following: conjunctival redness, conjunctival edema, lacrimal caruncle edema, eyelid redness, or eyelid edema. A device for predicting eye-related symptoms.

12. Each of the individual judgment values ​​includes a statement indicating that the corresponding individual judgment value is obtained from an ophthalmologist with more than 15 years of experience. The eye-related symptom prediction device according to claim 11.

Images