Method, system, and computer-readable recording medium for determination of arthritis grade using plurality of artificial neural models

Abstract

The present invention relates to a method, a system, and a computer-readable recording medium for determination of an arthritis grade using a plurality of artificial neural models, wherein a first model corresponding to a CNN-based artificial neural network model and a third model corresponding to a transformer-based artificial neural network model are trained through training data corresponding to X-ray images labeled in a first manner of labeling with a low arthritis grade, a second model corresponding to a CNN-based artificial neural network model and a fourth model corresponding to a transformer-based artificial neural network model are trained through training data corresponding to X-ray images labeled in a second manner of labeling with a high arthritis grade, and the arthritis grade for an X-ray image is determined using the first model, the second model, the third model, and the fourth model to diagnose the arthritis grade while implementing a process in which medical staff performs overall / local determination and optimistic / pessimistic determination of the X-ray image at an actual medical site.

Description

Method, system, and computer-readable recording medium for determining arthritis grade using multiple artificial neural models

[0001] The present invention relates to a method, system, and computer-readable recording medium for determining arthritis grade using a plurality of artificial neural models, wherein a first model corresponding to a CNN-based artificial neural network model and a third model corresponding to a Transformer-based artificial neural network model are trained using training data corresponding to an X-ray image labeled by a first method in which the arthritis grade is labeled low, and a second model corresponding to a CNN-based artificial neural network model and a fourth model corresponding to a Transformer-based artificial neural network model are trained using training data corresponding to an X-ray image labeled by a second method in which the arthritis grade is labeled high, and the arthritis grade for an X-ray image is determined using the first, second, third, and fourth models, thereby diagnosing the arthritis grade while implementing the process of medical personnel making overall / local judgments and optimistic / pessimistic judgments regarding the X-ray image in an actual medical setting.

[0002] Arthritis refers to a degenerative disease in which the joint cartilage deteriorates due to damage to shock-absorbing cartilage caused by factors such as aging or excessive exercise. Meanwhile, as the elderly population rapidly increases, the number of patients suffering from arthritis is also rising sharply; however, due to a shortage of medical personnel, the process of diagnosing and treating arthritis takes a considerable amount of time. Specifically, the diagnosis of arthritis can be based on complex judgments derived from various data and perspectives. For instance, even when viewing the same X-ray image, one medical professional may assess the arthritis grade as poor, while another may assess it as better.

[0003] Alternatively, medical professionals may diagnose arthritis by analyzing the patient's joint X-ray image as a whole, or conversely, by analyzing a specific part of the X-ray image locally.

[0004] Meanwhile, as a method for rapidly diagnosing arthritis, a method using artificial intelligence technology has recently been proposed. Specifically, by using a deep learning-based trained artificial intelligence model, the grade of arthritis can be automatically diagnosed from a patient's X-ray image.

[0005] However, conventional artificial intelligence-based arthritis grade judgment technology could not reflect both optimistic and pessimistic judgments regarding the same X-ray image, nor could it reflect both overall and local judgments; consequently, it was not possible to implement a judgment process similar to how medical professionals determine arthritis grades in actual clinical settings.

[0006] (Patent Document 0001) Korean Registered Patent KR 10-2414601B1 “Machine learning-based method for deriving bone density for hip fracture diagnosis and bone density derivation program using the same”

[0007] The present invention aims to provide a method, system, and computer-readable recording medium for determining arthritis grade using a plurality of artificial neural models, wherein a first model corresponding to a CNN-based artificial neural network model and a third model corresponding to a Transformer-based artificial neural network model are trained using training data corresponding to an X-ray image labeled by a first method in which the arthritis grade is labeled low, and a second model corresponding to a CNN-based artificial neural network model and a fourth model corresponding to a Transformer-based artificial neural network model are trained using training data corresponding to an X-ray image labeled by a second method in which the arthritis grade is labeled high, and the arthritis grade for the X-ray image is determined using the first, second, third, and fourth models, thereby diagnosing the arthritis grade while implementing the process of medical personnel making overall / local judgments and optimistic / pessimistic judgments regarding the X-ray image in an actual medical setting.

[0008] To solve the above problem, a method for determining arthritis grade performed in a computing system comprising one or more processors and one or more memories comprises: a first judgment information derivation step of preprocessing an X-ray image including a joint region and inputting the preprocessed X-ray image into a first model including an artificial neural network to derive first judgment information regarding arthritis grade; and a second judgment information derivation step of preprocessing an X-ray image including a joint region and inputting the preprocessed X-ray image into a second model including an artificial neural network to derive second judgment information regarding arthritis grade. The present invention provides a method for determining arthritis grade, comprising: a final judgment step for deriving final judgment information regarding arthritis grade based on comprehensive judgment information including the first judgment information and the second judgment information; wherein the first model is a deep learning-based artificial neural network model learned by training data labeled by a first method, and the second model is a deep learning-based artificial neural network model learned by training data labeled by a second method; for the same training data, which is an X-ray image, the first method is a method in which the labeling information regarding arthritis grade is labeled lower than the second method, and the arthritis grade is numerically expressed in proportion to or inversely proportion to the severity of arthritis.

[0009] In one embodiment of the present invention, the labeling information for the training data prior to being labeled by the first method or the second method is determined as a one-hot vector in which, in the stepwise classification of arthritis grades, the arthritis grade corresponding to the ground truth value of the training data is determined to be 1, and the arthritis grade not corresponding to the ground truth value is determined to be 0; the first method is a method of inputting an arbitrary number lower than 1 to an arthritis grade lower than the arthritis grade corresponding to 1 in the labeling information in the form of a one-hot vector, and the second method is a method of inputting an arbitrary number lower than 1 to an arthritis grade higher than the arthritis grade corresponding to 1 in the labeling information of the training data.

[0010] In one embodiment of the present invention, the first model may be trained such that there is a probability that the arthritis grade is determined to be lower than the arthritis grade corresponding to the ground truth value of the training data, and the second model may be trained such that there is a probability that the arthritis grade is determined to be higher than the arthritis grade corresponding to the ground truth value of the training data.

[0011] In one embodiment of the present invention, the first model and the second model are trained with training data in which labeling information is assigned using either the first method or the second method, with only the labeling method differing, while the first judgment information and the second judgment information may include information related to numerical values ​​that predict the input preprocessed X-ray image corresponds to each of the plurality of arthritis grades.

[0012] In one embodiment of the present invention, the first model comprises: a plurality of deep learning-based backbone neural network blocks that receive feature information output from the preprocessed X-ray image or another backbone neural network block of a previous stage and output feature information at that stage; a plurality of deep learning-based information selection modules that receive feature information output from the plurality of backbone neural network blocks and output selection information related to the judgment of arthritis grade in the corresponding feature information; and an integration module that receives information including a plurality of selection information output from the plurality of information selection modules and outputs first judgment information regarding the arthritis grade, wherein the backbone neural network blocks and the information selection modules may be artificial neural networks that compress data in the same way.

[0013] In one embodiment of the present invention, the arthritis grade determination method further comprises: a third judgment information derivation step of preprocessing an X-ray image including a joint area and inputting the preprocessed X-ray image into a third model including an artificial neural network to derive third judgment information regarding the arthritis grade; and a fourth judgment information derivation step of preprocessing an X-ray image including a joint area and inputting the preprocessed X-ray image into a fourth model including an artificial neural network to derive fourth judgment information regarding the arthritis grade; wherein the judgment comprehensive information further comprises third judgment information and fourth judgment information, and the third model and the fourth model may be artificial neural networks that process or compress data in a manner different from that of the first model and the second model.

[0014] In one embodiment of the present invention, the first model and the second model are a CNN (Convolutional Neural Network) or a Transformer-based artificial neural network model, and the third model and the fourth model may be a Transformer-based artificial neural network model when the first model and the second model are CNN-based artificial neural network models, and a CNN-based artificial neural network model when the first model and the second model are Transformer-based artificial neural network models.

[0015] In one embodiment of the present invention, the third model may be a deep learning-based artificial neural network model trained on training data labeled by the first method, and the fourth model may be a deep learning-based artificial neural network model trained on training data labeled by the second method.

[0016] To solve the above problem, an arthritis grade determination system comprising one or more processors and one or more memories, comprising: a first judgment information derivation unit that preprocesses an X-ray image including a joint region and inputs the preprocessed X-ray image into a first model including an artificial neural network to derive first judgment information regarding the arthritis grade; and a second judgment information derivation unit that preprocesses an X-ray image including a joint region and inputs the preprocessed X-ray image into a second model including an artificial neural network to derive second judgment information regarding the arthritis grade. and a final judgment unit that derives final judgment information regarding the arthritis grade based on comprehensive judgment information including the first judgment information and the second judgment information; wherein the first model is a deep learning-based artificial neural network model learned by training data labeled by the first method, and the second model is a deep learning-based artificial neural network model learned by training data labeled by the second method, and for the same training data which is an X-ray image, the first method is a method in which the labeling information regarding the arthritis grade is labeled lower than the second method, and the arthritis grade can be numerically expressed in proportion to or inversely proportion to the severity of arthritis.

[0017] To solve the above problem, a computer-readable recording medium comprising one or more processors and one or more memories and performing a method for determining arthritis grade, wherein the computer-readable recording medium stores instructions for performing the following steps, and the following steps include: a first judgment information derivation step of preprocessing an X-ray image including a joint region and inputting the preprocessed X-ray image into a first model including an artificial neural network to derive first judgment information regarding arthritis grade; and a second judgment information derivation step of preprocessing an X-ray image including a joint region and inputting the preprocessed X-ray image into a second model including an artificial neural network to derive second judgment information regarding arthritis grade. The present invention provides a computer-readable recording medium comprising: a final judgment step for deriving final judgment information regarding an arthritis grade based on comprehensive judgment information including the first judgment information and the second judgment information; wherein the first model is a deep learning-based artificial neural network model learned by training data labeled by a first method, and the second model is a deep learning-based artificial neural network model learned by training data labeled by a second method, and for the same training data which is an X-ray image, the first method is a method in which the labeling information regarding the arthritis grade is labeled lower than the second method, and the arthritis grade is numerically expressed in proportion to or inversely proportion to the severity of arthritis.

[0018] According to one embodiment of the present invention, an artificial intelligence model can be trained using training data labeled with a lower arthritis grade than the ground truth for the arthritis grade of an X-ray image, thereby enabling the implementation of an optimistic judgment regarding the arthritis grade.

[0019] According to one embodiment of the present invention, an artificial intelligence model can be trained using training data labeled with a higher arthritis grade than the ground truth for the arthritis grade of an X-ray image, thereby enabling the implementation of an intention to judge the arthritis grade pessimistically.

[0020] According to one embodiment of the present invention, the grade of arthritis can be determined by locally analyzing a portion of an X-ray image using a CNN-based artificial intelligence inference model.

[0021] According to one embodiment of the present invention, the grade of arthritis can be determined by analyzing an X-ray image as a whole using a transformer-based artificial intelligence inference model.

[0022] According to one embodiment of the present invention, the grade of arthritis in an X-ray image can be determined based on feature information derived by an artificial intelligence inference model that analyzes an X-ray image locally from an optimistic perspective, an artificial intelligence inference model that analyzes an X-ray image locally from a pessimistic perspective, an artificial intelligence inference model that analyzes an X-ray image overall from an optimistic perspective, and an artificial intelligence inference model that analyzes an X-ray image overall from a pessimistic perspective.

[0023] According to one embodiment of the present invention, even if the arthritis grade of a patient determined using an artificial intelligence inference model is incorrect, the incorrect answer can be made to have a result similar to the actual arthritis grade of the patient. For example, if the actual arthritis grade of a patient is level 4, the probability that the artificial intelligence inference model determines the patient's arthritis grade as level 3 may be higher than the probability that it determines it as level 0.

[0024] Figures 1a, 1b, and 1c illustrate the process by which medical staff determine the grade of arthritis using X-ray images in an actual medical setting.

[0025] FIG. 2 illustrates the components of an arthritis grade determination system according to one embodiment of the present invention.

[0026] FIGS. 3a and FIGS. 3b illustrate training data labeled by a first method and a second method according to an embodiment of the present invention.

[0027] FIGS. 4a and 4b illustrate matters related to the learning and inference of a first model and a second model according to an embodiment of the present invention.

[0028] FIG. 5 illustrates the components of a first model according to one embodiment of the present invention.

[0029] FIG. 6 illustrates matters related to the information processing method of the first model and the second model according to one embodiment of the present invention.

[0030] FIGS. 7a and 7b illustrate matters related to the learning and inference of the third and fourth models according to an embodiment of the present invention.

[0031] FIG. 8 illustrates matters related to the information processing method of the third model and the fourth model according to an embodiment of the present invention.

[0032] FIG. 9 illustrates the judgment method of the first model and the third model according to one embodiment of the present invention.

[0033] FIG. 10 illustrates details regarding a method for determining arthritis grade according to one embodiment of the present invention.

[0034] FIG. 11 illustrates a pretreatment process according to one embodiment of the present invention.

[0035] FIG. 12 schematically illustrates the internal configuration of a computing device according to one embodiment of the present invention.

[0036] The present invention relates to a method, system, and computer-readable recording medium for determining arthritis grade using a plurality of artificial neural models, wherein a first model corresponding to a CNN-based artificial neural network model and a third model corresponding to a Transformer-based artificial neural network model are trained using training data corresponding to an X-ray image labeled by a first method in which the arthritis grade is labeled low, and a second model corresponding to a CNN-based artificial neural network model and a fourth model corresponding to a Transformer-based artificial neural network model are trained using training data corresponding to an X-ray image labeled by a second method in which the arthritis grade is labeled high, and the arthritis grade for the X-ray image is determined using the first, second, third, and fourth models, thereby diagnosing the arthritis grade while implementing the process of medical personnel making overall / local judgments and optimistic / pessimistic judgments regarding the X-ray image in an actual medical setting.

[0037] Hereinafter, various embodiments and / or aspects are disclosed with reference to the drawings. For illustrative purposes, numerous specific details are disclosed in the following description to aid in a general understanding of one or more aspects. However, it will also be recognized by those skilled in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings describe specific exemplary aspects of one or more aspects in detail. However, these aspects are exemplary, and some of the various methods in the principles of the various aspects may be used, and the description is intended to include all such aspects and their equivalents.

[0038] In addition, various aspects and features will be presented by a system that may include multiple devices, components and / or modules, etc. It should also be understood and recognized that various systems may include additional devices, components and / or modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in relation to the drawings.

[0039] As used herein, terms such as “example,” “example,” “aspect,” “example,” etc., may not be interpreted as implying that any aspect or design described is superior or more advantageous than other aspects or designs. Terms used below, such as “part,” “component,” “module,” “system,” “interface,” etc., generally refer to computer-related entities and may, for example, refer to hardware, a combination of hardware and software, or software.

[0040] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that the relevant feature and / or component is present, but not to exclude the presence or addition of one or more other features, components and / or groups thereof.

[0041] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0042] Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0043]

[0044] Figures 1a, 1b, and 1c illustrate the process by which medical staff determine the grade of arthritis using X-ray images in an actual medical setting.

[0045]

[0046] As illustrated in Fig. 1a, the process of medical staff determining a patient's arthritis grade can be based on a comprehensive judgment derived from various information. Specifically, medical staff can determine the patient's arthritis grade by analyzing X-ray images of the patient's joint area (preferably the knee joint).

[0047] Such arthritis grades can be classified into stages using the KL grade. Specifically, the KL grade is divided into stages 0, 1, 2, 3, and 4, and the higher the stage, the worse the arthritis grade.

[0048] For example, Stage 0 corresponds to a normal state with no particular problems on X-ray images, Stage 1 corresponds to a state where the stenosis of the joint space is ambiguous or there may be bone spur formation, Stage 2 corresponds to a state where there is a possibility of stenosis of the joint space or there is definite bone spur, Stage 3 corresponds to a state where there is stenosis of the joint space and multiple severe bone spurs, and there may be subchondral sclerosis and bone deformation, and Stage 4 corresponds to a state where the stenosis of the joint space is severe, the bone spurs are large, and there is severe subchondral sclerosis and bone deformation.

[0049] Medical staff can analyze the patient's X-ray images from various perspectives and diagnose the patient's arthritis grade as KL grade 0, 1, 2, 3, or 4.

[0050] Specifically, medical staff can determine the patient's arthritis grade by analyzing the joint space, cartilage condition, bone spurs, and bone curvature while viewing the X-ray image as a whole. Alternatively, medical staff can determine the patient's arthritis grade by focusing on specific parts of the X-ray image (for example, bone spurs or cysts locally occurring in a specific area, or the joint space in each area by dividing the joint into medial and lateral sides).

[0051] As such, even with the same X-ray image, the patient's arthritis grade may be determined differently depending on the way the medical staff views (diagnoses). Or, even with the same X-ray image, the patient's arthritis grade may be determined differently depending on the medical staff.

[0052]

[0053] In addition, medical staff can determine the patient's arthritis grade by considering not only the patient's X-ray images but also additional information related to the patient (e.g., age, medical history, family history).

[0054]

[0055] As illustrated in Fig. 1b, the grade of arthritis may be determined differently depending on the medical staff's disposition. For example, even with the same X-ray image, a doctor who diagnoses the patient's arthritis grade optimistically may diagnose the patient's arthritis grade as low. On the other hand, a doctor who diagnoses the patient's arthritis grade pessimistically may diagnose the patient's arthritis grade as high.

[0056] For example, regarding the same X-ray image, an optimistic doctor might judge the arthritis grade to be level 3, whereas a pessimistic doctor might judge it to be level 4, which is worse than level 3. In other words, a pessimistic doctor can be understood as a doctor who tends to judge the joint condition as poor, in contrast to an optimistic doctor.

[0057]

[0058] As such, when medical professionals analyze a patient's X-ray images in actual medical practice to determine the grade of arthritis, the grade may be determined differently depending on the analysis method (whether to analyze the X-ray image holistically or locally), the perspective (whether to make an optimistic or pessimistic judgment), and the way additional information about the patient (patient's age, medical history, family history, etc.) is interpreted.

[0059] Consequently, in actual medical settings, the arthritis grade for a patient can be finally determined by comprehensively considering arthritis grades judged according to various analysis methods and perspectives, and the present invention can implement a realistic and highly accurate arthritis grade determination system by determining the patient's arthritis grade in a manner similar to the process carried out in actual medical settings.

[0060]

[0061] Meanwhile, as illustrated in FIG. 1c, the arthritis grade can be numerically expressed in proportion to or inversely proportional to the severity of arthritis. Specifically, the arthritis grade is classified into KL grades ranging from 0 to 4, and it can be understood that the higher the value, the more severe the arthritis. Additionally, the arthritis grades may be relatively highly correlated with each other in terms of numerical values.

[0062] For example, the best condition, stage 0, and the worst condition, stage 4, may have the least correlation with each other. That is, when an X-ray image of a patient corresponding to stage 0 is given to medical staff, the probability that the medical staff will determine the arthritis grade as stage 1 may be higher than the probability of stage 4.

[0063] Conversely, stages 2 and 3 may be highly correlated (compared to the correlation between stages 0 and 4). That is, when an X-ray image of a patient corresponding to stage 3 is given to medical staff, the probability that the medical staff will determine the arthritis grade as stage 2 or 4 (numerically close to stage 3) may be higher than the probability that it will be stage 0 or 4 (numerically far from stage 3).

[0064]

[0065] As such, the smaller the difference in the numerical values ​​(stages) of the arthritis grades classified by KL grade, the higher the correlation between the corresponding stages, and the larger the difference in the numerical values ​​(stages) of the arthritis grades, the lower the correlation between the corresponding stages.

[0066]

[0067] The arthritis grade determination system of the present invention described below can determine a patient's arthritis grade while reflecting the characteristics of the diagnostic process and arthritis grades that take place in actual medical settings as described in FIGS. 1a, 1b, and 1c.

[0068]

[0069] FIG. 2 illustrates the components of an arthritis grade determination system according to one embodiment of the present invention.

[0070]

[0071] As illustrated in FIG. 2, a method for determining arthritis grade performed in a computing system comprising one or more processors and one or more memories, comprising: a first judgment information derivation step of preprocessing an X-ray image including a joint region and inputting the preprocessed X-ray image into a first model (20) including an artificial neural network to derive first judgment information regarding arthritis grade; and a second judgment information derivation step of preprocessing an X-ray image including a joint region and inputting the preprocessed X-ray image into a second model (30) including an artificial neural network to derive second judgment information regarding arthritis grade. and a final judgment step for deriving final judgment information for arthritis grade based on comprehensive judgment information including the first judgment information and the second judgment information; wherein the first model (20) is a deep learning-based artificial neural network model learned by training data labeled by the first method, and the second model (30) is a deep learning-based artificial neural network model learned by training data labeled by the second method, and for the same training data which is an X-ray image, the first method is a method in which the labeling information for the arthritis grade is labeled lower than the second method, and the arthritis grade can be numerically expressed in proportion to or inversely proportion to the severity of arthritis.

[0072]

[0073] Specifically, the arthritis grade evaluation system of the present invention can receive an X-ray image including a joint area and derive judgment information regarding the arthritis grade of the X-ray image, and the arthritis grade can be expressed as a stepwise numerical value classified as KL grade.

[0074] Preferably, the judgment information can be derived as any one of KL grades from 0 to 4. For example, the judgment information can be derived as the X-ray image corresponding to KL grade 2.

[0075] Alternatively, judgment information can be derived as a probability for each of the KL grades from 0 to 4. For example, judgment information can be derived as a probability that the X-ray image corresponds to each of the KL grades 0, 1, 2, 3, and 4.

[0076]

[0077] An arthritis grade evaluation system may include a preprocessing unit (1) for preprocessing an X-ray image, and the preprocessing unit (1) may include a resizing unit (10) for resizing the size of the X-ray image, a brightness control unit (11) for adjusting the brightness of the X-ray image, a histogram extraction unit (12) for extracting a histogram from the X-ray image, and an ORB extraction unit (13) for extracting ORB (Oriented FAST and Rotated BRIEF) from the X-ray image.

[0078]

[0079] The preprocessing unit (1), which includes such components, can process the X-ray image into a form that can be input to each artificial intelligence inference model. For example, the preprocessing unit (1) can adjust the size of the X-ray image to an image size that can be analyzed by each of the first model (20), second model (30), third model (40), and fourth model (50), or can extract histograms and ORBs necessary for determining the arthritis grade.

[0080] In this way, the preprocessing unit (1) can process the X-ray image into a form optimized for each of the first model (20), second model (30), third model (40), and fourth model (50).

[0081]

[0082] The X-ray image preprocessed by the preprocessing unit (1) can be input to each of the first model (20), the second model (30), the third model (40), and the fourth model (50). Specifically, the first model (20), the second model (30), the third model (40), and the fourth model (50) may be deep learning-based learned artificial intelligence inference models.

[0083]

[0084] Preferably, the first model (20) and the second model (30) may be artificial intelligence inference models having the same attributes as each other and different attributes from the third model (40) and the fourth model (50). For example, as shown in FIG. 2, the first model (20) and the second model (30) may be artificial intelligence inference models based on a Convolutional Neural Network (CNN), and the third model (40) and the fourth model (50) may be Transformer-based artificial intelligence inference models.

[0085]

[0086] Additionally, the first model (20) and the second model (30) are artificial intelligence inference models trained with training data labeled in different ways, and the third model (40) and the fourth model (50) may be artificial intelligence inference models trained with training data labeled in different ways.

[0087] Specifically, the first model (20) and the third model (40) are artificial intelligence inference models trained with training data labeled in the first method, and the second model (30) and the fourth model (50) may be artificial intelligence inference models trained with training data labeled in the second method.

[0088] In one embodiment of the present invention, by training the same artificial intelligence inference model with training data labeled in different first or second methods, an artificial intelligence inference model that optimistically judges the arthritis grade and an artificial intelligence inference model that pessimistically judges the grade can be implemented. A detailed explanation thereof will be provided later.

[0089]

[0090] As such, the arthritis grade evaluation system may include multiple (preferably four) artificial intelligence inference models trained with training data labeled in different ways (first way or second way) and having different attributes (CNN or Transformer).

[0091] Meanwhile, each artificial intelligence inference model can derive judgment information regarding the arthritis grade. Specifically, the first model (20) receives a preprocessed X-ray image and derives first judgment information regarding the arthritis grade, the second model (30) receives a preprocessed X-ray image and derives second judgment information regarding the arthritis grade, the third model (40) receives a preprocessed X-ray image and derives third judgment information regarding the arthritis grade, and the fourth model (50) receives a preprocessed X-ray image and derives fourth judgment information regarding the arthritis grade.

[0092] In addition, each of the first judgment information, second judgment information, third judgment information, and fourth judgment information can be derived as any one of the KL grades from 0 to 4 or as a probability for each grade.

[0093]

[0094] Meanwhile, in one embodiment of the present invention, the arthritis grade evaluation system may be implemented in a form including two or more of the first model (20), the second model (30), the third model (40), and the fourth model (50). For example, the arthritis grade evaluation system may be implemented in a form including only the first model (20) and the second model (30).

[0095]

[0096] The final judgment unit (6) can derive final judgment information for the arthritis grade based on comprehensive judgment information including first judgment information, second judgment information, third judgment information, and fourth judgment information derived from each of the first model (20), second model (30), third model (40), and fourth model (50).

[0097] For example, the final judgment unit (6) can derive final judgment information based on the first judgment information, second judgment information, third judgment information, and fourth judgment information using a random forest algorithm.

[0098]

[0099] As a result, the arthritis grade evaluation system can derive information about which stage of KL grade the X-ray image corresponds to for the input X-ray image.

[0100]

[0101] FIGS. 3a and FIGS. 3b illustrate training data labeled by a first method and a second method according to an embodiment of the present invention.

[0102]

[0103] As illustrated in FIGS. 3a and 3b, the labeling information for the training data prior to being labeled by the first or second method is determined as a one-hot vector in which, in the stepwise separated arthritis grades, the arthritis grade corresponding to the ground truth value of the training data is determined to be 1, and the arthritis grade not corresponding to the ground truth value is determined to be 0. The first method is a method of inputting an arbitrary number lower than 1 to an arthritis grade lower than the arthritis grade corresponding to 1 in the labeling information in the form of a one-hot vector, and the second method is a method of inputting an arbitrary number lower than 1 to an arthritis grade higher than the arthritis grade corresponding to 1 in the labeling information of the training data.

[0104]

[0105] In the following, the training data may preferably be X-ray images.

[0106]

[0107] As illustrated in FIG. 3a, the labeling information for training data prior to being labeled by the first or second method can be determined according to the ground truth value of the training data. Specifically, in FIG. 3a, if there is training data with a ground truth value of KL grade level 1, the labeling information for the training data can be determined such that the value for level 1 corresponding to the ground truth value is 1, and the values ​​for levels 0, 2, 3, and 4 not corresponding to the ground truth value are 0.

[0108] In other words, in Fig. 3a, the training data is an X-ray image of a patient whose actual KL grade is level 1, and the labeling information can be understood as being labeled such that the probability of the training data corresponding to levels 0, 2, 3, and 4 is proportional to 0, and the probability of corresponding to level 1 is proportional to 1.

[0109]

[0110] In this way, the labeling information may be in the form of a one-hot vector in which, in the stepwise classification of arthritis grades, the arthritis grade corresponding to the ground truth value is determined to be 1, and the arthritis grade not corresponding to the ground truth value is determined to be 0.

[0111]

[0112] Meanwhile, in one embodiment of the present invention, the artificial intelligence inference model (first model (20), second model (30), third model (40), and fourth model (50)) is not trained with training data in which the labeling information is labeled in this manner, but the artificial intelligence inference model can be trained with training data after the labeling information has been artificially processed in the first method or the second method.

[0113]

[0114] As illustrated in FIG. 3b, the first method may be a method of artificially manipulating labeling information to lower the arthritis grade. For example, in FIG. 3b, the first method may be a method of arbitrarily modifying the value of an arthritis grade (level 0) lower than an arthritis grade (level 1) with a value of 1 in the labeling information from 0 to an arbitrary number (0.95). Conversely, in the first method, the values ​​of arthritis grades (levels 2, 3, and 4) higher than an arthritis grade (level 1) with a value of 1 are not modified from 0.

[0115] Preferably, the first method may be a method of modifying the value of the closest arthritis grade level that is lower than the arthritis grade level determined to be 1 in the labeling information according to the ground truth value of the training data to a value lower than 1.

[0116] That is, in the first method, when the stage of the arthritis grade determined as 1 in the labeling information is stage 2, the value of stage 1, which is the closest stage smaller than stage 2, is modified to 0.95, which is an arbitrary number smaller than 1, and the value of stage 0, which is the closest stage lower than stage 2, is not modified and can be determined as 0.

[0117]

[0118] In other words, in FIG. 3b, the training data is an X-ray image of a patient whose actual KL grade is 1, and the labeling information of the training data labeled by the first method can be understood as being labeled such that the probability of the training data corresponding to 1 is proportional to 1, the probability of corresponding to 0 (lower than 1) is proportional to 0.95, and the probability of corresponding to 2, 3, and 4 is proportional to 0.

[0119]

[0120] As illustrated in FIG. 3b, the second method may be a method of artificially processing labeling information to increase the arthritis grade. For example, in FIG. 3b, the second method may be a method of arbitrarily modifying the value of an arthritis grade (level 2) higher than an arthritis grade (level 1) with a value of 1 in the labeling information from 0 to an arbitrary number (0.95). Conversely, in the second method, the value of an arthritis grade (level 0) lower than an arthritis grade (level 1) with a value of 1 is not modified from 0.

[0121] Preferably, the second method may be a method of modifying the value of the arthritis grade level closest to and higher than the arthritis grade level determined to be 1 in the labeling information according to the ground truth value of the training data to a value lower than 1.

[0122] That is, in the second method, when the stage of the arthritis grade determined as 1 in the labeling information is stage 2, the value of stage 3, which is the closest stage higher than stage 2, is modified to 0.95, which is an arbitrary number less than 1, and the value of stage 4, which is the closest stage higher than stage 2, is not modified and can be determined as 0.

[0123]

[0124] In other words, in FIG. 3b, the training data is an X-ray image of a patient whose actual KL grade is 1, and the labeling information of the training data labeled by the second method can be understood as being labeled such that the probability of the training data corresponding to 1 is proportional to 1, the probability of corresponding to 2, which is higher than 1, is proportional to 0.95, and the probability of corresponding to 0, 3, and 4 is proportional to 0, respectively.

[0125]

[0126] FIGS. 4a and FIGS. 4b illustrate matters related to the learning and inference of a first model (20) and a second model (30) according to an embodiment of the present invention.

[0127]

[0128] As illustrated in FIGS. 4a and 4b, the first model (20) is trained such that there is a probability that the arthritis grade is determined to be lower than the arthritis grade corresponding to the ground truth value of the training data, and the second model (30) is trained such that there is a probability that the arthritis grade is determined to be higher than the arthritis grade corresponding to the ground truth value of the training data.

[0129] In addition, the first model (20) and the second model (30) are trained with training data in which labeling information is assigned using either the first method or the second method, with only the labeling method differing, while the first judgment information and the second judgment information may include information related to a numerical value that predicts the input preprocessed X-ray image corresponds to each of the multiple arthritis grades.

[0130]

[0131] As illustrated in FIG. 4a, the first model (20) can be trained with training data labeled in the first method (where the labeling information for the arthritis grade is labeled lower than in the second method), and conversely, the second model (30) can be trained with training data labeled in the second method (where the labeling information for the arthritis grade is labeled higher than in the first method).

[0132] Specifically, the first model (20) and the second model (30) can be understood as being the same artificial intelligence inference model, but trained using different training data. Preferably, the first model (20) and the second model (30) can be understood as being trained using training data in which only the labeling information on the same X-ray image is labeled differently.

[0133]

[0134] As described above, the first model (20) and the second model (30) are identical artificial intelligence inference models, and their internal structures and data processing methods may be identical. Specifically, if the first model (20) is a CNN-based artificial intelligence inference model, the second model (30) may also be a CNN-based artificial intelligence inference model. Conversely, if the first model (20) is a Transformer-based artificial intelligence inference model, the second model (30) may also be a Transformer-based artificial intelligence inference model.

[0135] In addition, the internal structure of the first model (20) and the second model (30) may be identical, and a detailed explanation thereof will be provided later.

[0136]

[0137] As illustrated in FIG. 4b, the first model (20) has a high probability of determining the arthritis grade in the X-ray image as low (compared to the second model (30)), and conversely, the second model (30) has a high probability of determining the arthritis grade in the X-ray image as high (compared to the first model (20)).

[0138] Specifically, an X-ray image is input to the first model (20) that has completed learning and first judgment information is output, and an X-ray image is input to the second model (30) that has completed learning and second judgment information is output, and at this time, there may be a higher probability that the arthritis grade in the second judgment information is lower than the arthritis grade in the first judgment information.

[0139] For example, for the same X-ray image in FIG. 4b where the KL grade is level 2, the first model (20) can determine with a certain probability that the X-ray image is level 1, which is lower than level 2.

[0140]

[0141] Preferably, the first model (20) can determine the arthritis grade to be lower than the actual arthritis grade in the X-ray image, but even if the arthritis grade is determined to be lower than the actual arthritis grade, the probability of determining it as ‘the arthritis grade that is lower and closest (with the smallest difference) than the actual arthritis grade’ may be high, or the probability of determining it as ‘the arthritis grade that is lower and not closest than the actual arthritis grade’ or ‘the arthritis grade that is higher than the actual arthritis grade’ may be high.

[0142] For example, when an X-ray image with an arthritis grade of 2 is input into the first model (20), even if the first model (20) determines the arthritis grade to be lower than 2, there is a high probability that it will determine it as ‘1st grade, which is closest to 2nd grade’, or ‘0th grade, which is a greater difference from 2nd grade than 1st grade’, or ‘3rd grade or 4th grade, which is higher than 2nd grade’.

[0143] This is because, as previously mentioned, the first model (20) is trained with training data labeled in the first method in which the value of the arthritis grade corresponding to the ground truth value is 1, and the value of the arthritis grade closest to the arthritis grade corresponding to the ground truth value is determined to be an arbitrary value lower than 1.

[0144] In other words, conceptually, the first model (20) may be an artificial intelligence inference model that optimistically judges the arthritis grade (judging the arthritis grade better than the actual arthritis grade).

[0145]

[0146] Additionally, the second model (30) may have a higher probability of determining the arthritis grade in the X-ray image (compared to the first model (20)). Specifically, the arthritis grade in the second judgment information output by the second model (30) after training may be higher than the arthritis grade in the first judgment information output by the first model (20).

[0147] For example, for an X-ray image with a KL grade of 2, the second model (30) can determine with a certain probability that the X-ray image is of a higher grade than 2, which is 3.

[0148] Preferably, the second model (30) can determine the arthritis grade to be higher than the actual arthritis grade in the X-ray image, but even if the arthritis grade is determined to be higher than the actual arthritis grade, the probability of determining it as ‘the arthritis grade that is higher than the actual arthritis grade and closest (with the smallest difference)’ may be high, or the probability of determining it as ‘the arthritis grade that is higher than the actual arthritis grade and not closest’ or ‘the arthritis grade that is lower than the actual arthritis grade’ may be high.

[0149] For example, when an X-ray image with an arthritis grade of 2 is input to the second model (30), even if the second model (30) determines the arthritis grade to be higher than 2, there is a high probability that it will determine it as ‘3rd grade, which is closest to 2nd grade’, or ‘4th grade, which is a greater difference from 2nd grade than 3rd grade’, or ‘0th grade or 1st grade, which is lower than 2nd grade’.

[0150] This is because, as previously mentioned, the second model (30) is trained with training data labeled in a second method in which the value of the arthritis grade corresponding to the ground truth value is 1, and the value of the arthritis grade closest to the arthritis grade corresponding to the ground truth value is determined to be an arbitrary value lower than 1.

[0151] In other words, conceptually, the second model (30) may be an artificial intelligence inference model that judges the arthritis grade pessimistically (judging the arthritis grade worse than the actual arthritis grade).

[0152]

[0153] In this way, the present invention can derive final judgment information regarding the patient's arthritis grade by comprehensively utilizing the first judgment information and the second judgment information output by inputting the same X-ray image into the first model (20) and the second model (30) that have completed learning.

[0154] Specifically, the present invention allows for a comprehensive determination of the arthritis grade based on X-ray images using an artificial intelligence inference model, while considering both optimistic and pessimistic perspectives.

[0155] Furthermore, in determining the arthritis grade of an X-ray image using an artificial intelligence inference model, the present invention allows the difference from the actual correct answer to be small even if the judgment information regarding the arthritis grade is incorrect. That is, even if the judgment information regarding the arthritis grade is output as an incorrect answer in the present invention, the output can be similar to the severity of the patient's arthritis.

[0156] For example, in the present invention, it is preferable that the final judgment information for an X-ray image with an arthritis grade of 2 be output as 2, but even if it is not output as 2, the probability of outputting as 1 or 3, which is close to 2, may be greater than the probability of outputting as 0 or 4, which is not close to 2.

[0157]

[0158] FIG. 5 illustrates the components of a first model (20) according to one embodiment of the present invention.

[0159]

[0160] As illustrated in FIG. 5, the first model (20) comprises: a plurality of deep learning-based backbone neural network blocks that receive feature information output from the preprocessed X-ray image or another backbone neural network block of the previous stage and output feature information at that stage; a plurality of deep learning-based information selection modules that receive feature information output from the plurality of backbone neural network blocks and output selection information related to the judgment of arthritis grade in the corresponding feature information; and an integration module that receives information including a plurality of selection information output from the plurality of information selection modules and outputs first judgment information regarding the arthritis grade; wherein the backbone neural network blocks and the information selection modules may be artificial neural networks that compress data in the same way.

[0161]

[0162] As described above, the first model (20) and the second model (30) of the present invention are artificial intelligence inference models having substantially the same structure, and only the training data used to train the artificial intelligence inference models may be different.

[0163] That is, the second model (30) may also include the components of the first model (20) shown in FIG. 5. Below, the explanation will be based on the first model (20).

[0164]

[0165] The first model (20) may include a plurality of deep learning-based backbone neural network blocks that receive preprocessed X-ray images or feature information output from other backbone neural network blocks of the previous stage and output feature information at that stage.

[0166] Specifically, multiple backbone neural network blocks may have a hierarchical structure in which the feature output from the backbone neural network block of the preceding stage is input to the backbone neural network block of the subsequent stage.

[0167] For example, in FIG. 5, a preprocessed X-ray image is input to the first backbone neural network block (200.1) to output feature information F1, the output feature information F1 is input to the second backbone neural network block (200.2) to output feature information F2, and the output feature information F2 is input to the third backbone neural network block (200.3) to output feature information F3.

[0168] In one embodiment of the present invention, the feature information may be data in the form of a feature map that is expressed by recognizing shapes, patterns, edges, etc., in an X-ray image.

[0169]

[0170] The first model (20) may include multiple deep learning-based information selection modules that receive feature information output from multiple backbone neural network blocks and output selection information related to the determination of arthritis grade from the feature information.

[0171] Specifically, the information selection module can select data having specific patterns, edges, and textures from feature information and output selection information, and the selection information may be data related to arthritis grade.

[0172] For example, in FIG. 5, the first information selection module (201.1) distinguishes between selection information f1 related to the determination of arthritis grade and unnecessary information l1 that is not related to it in feature information F1, the second information selection module (201.2) distinguishes between selection information f2 related to the determination of arthritis grade and unnecessary information l2 that is not related to it in feature information F2, and the third information selection module (201.3) distinguishes between selection information f3 related to the determination of arthritis grade and unnecessary information l3 that is not related to it in feature information F3.

[0173]

[0174] In one embodiment of the present invention, the information selection module may be a model trained with unlabeled or inaccurately labeled training data to select a specific subset (e.g., information related to arthritis) from the input data.

[0175]

[0176] In one embodiment of the present invention, the information selection module may be composed of a number corresponding to the number of backbone neural network modules.

[0177]

[0178] The first model (20) may include an integrated module that receives information including multiple screening information output from multiple information screening modules and outputs first judgment information regarding the arthritis grade.

[0179] Specifically, the integration module can receive multiple selection information f1, f2, and f3 output from each of the multiple information selection modules and derive first judgment information regarding the arthritis grade.

[0180] In one embodiment of the present invention, one or more of the feature information F3 output from the backbone neural network block of the last stage and the X-ray image input to the backbone neural network block of the first stage may be input to the integration module.

[0181]

[0182] In one embodiment of the present invention, when the first model (20) is a CNN-based artificial intelligence inference model, the backbone neural network block may include one or more of a Convolutional Layer that recognizes and extracts features from input data, a Pooling Layer that reduces or summarizes features, and a Fully Connected Layer that calculates the final output.

[0183] In one embodiment of the present invention, when the first model (20) is a transformer-based artificial intelligence inference model, the backbone neural network block may include one or more of a Self-Attention block that calculates feature information by considering the relationship between input data and a Multi-Head Attention block that learns various patterns in parallel.

[0184]

[0185] Meanwhile, the third model (40) and the fourth model (50) described below may also include a plurality of backbone neural network blocks, a plurality of information selection modules, and an integration module.

[0186]

[0187] FIG. 6 illustrates matters related to the information processing method of the first model (20) and the second model (30) according to one embodiment of the present invention.

[0188]

[0189] As illustrated in FIG. 6, a plurality of backbone neural network blocks and a plurality of information selection modules constituting the first model (20) may be an artificial intelligence inference model having the same attributes. Alternatively, a plurality of backbone neural network blocks and a plurality of information selection modules may be an artificial intelligence inference model that processes data in the same way.

[0190] As described above, the first model (20) may be a CNN-based artificial intelligence inference model or a Transformer-based artificial intelligence inference model.

[0191] In this case, if the first model (20) is a CNN-based artificial intelligence inference model, the plurality of backbone neural network blocks and the plurality of information selection modules constituting the first model (20) may all be CNN-based artificial intelligence inference models.

[0192] Conversely, if the first model (20) is a transformer-based artificial intelligence inference model, the plurality of backbone neural network blocks and the plurality of information selection modules constituting the first model (20) may all be transformer-based artificial intelligence inference models.

[0193]

[0194] Also, as previously mentioned, since the second model (30) corresponds substantially to the same artificial intelligence inference model as the first model (20), the plurality of backbone neural network blocks (200.1, 200.2, 200.3, 300.1, 300.2, and 300.3) and the plurality of information selection modules (201.1, 201.2, 201.3, 301.1, 301.2, and 301.3) constituting the first model (20) and the second model (30) may be artificial intelligence inference models that process data in the same way.

[0195]

[0196] FIGS. 7a and 7b illustrate matters related to the learning and inference of the third model (40) and the fourth model (50) according to one embodiment of the present invention.

[0197]

[0198] As illustrated in FIGS. 7a and 7b, the arthritis grade determination method further comprises: a third judgment information derivation step of preprocessing an X-ray image including a joint area and inputting the preprocessed X-ray image into a third model (40) including an artificial neural network to derive third judgment information regarding the arthritis grade; and a fourth judgment information derivation step of preprocessing an X-ray image including a joint area and inputting the preprocessed X-ray image into a fourth model (50) including an artificial neural network to derive fourth judgment information regarding the arthritis grade; wherein the judgment comprehensive information further comprises third judgment information and fourth judgment information, and the third model (40) and the fourth model (50) may be artificial neural networks that process or compress data in a different way than the first model (20) and the second model (30).

[0199] In addition, the third model (40) may be a deep learning-based artificial neural network model trained on training data labeled in the first method, and the fourth model (50) may be a deep learning-based artificial neural network model trained on training data labeled in the second method.

[0200] In one embodiment of the present invention, the third judgment information derivation step and the fourth judgment information derivation step may be performed prior to the final judgment step. Alternatively, the third judgment information derivation step and the fourth judgment information derivation step may be included in the first judgment information derivation step or the second judgment information derivation step.

[0201]

[0202] The third model (40) and the fourth model (50) may be artificial intelligence inference models that determine the arthritis grade of an X-ray image using a different data processing method than the first model (20) and the second model (30).

[0203] Specifically, if the first model (20) and the second model (30) are CNN-based artificial neural network models, the third model (40) and the fourth model (50) may be Transformer-based artificial neural network models. Conversely, if the first model (20) and the second model (30) are Transformer-based artificial neural network models, the third model (40) and the fourth model (50) may be CNN-based artificial neural network models.

[0204]

[0205] As illustrated in FIG. 7a, the third model (40) can be trained with training data labeled in the first method (where the labeling information for the arthritis grade is labeled lower than in the second method), and conversely, the fourth model (50) can be trained with training data labeled in the second method (where the labeling information for the arthritis grade is labeled higher than in the first method).

[0206] Specifically, the third model (40) and the fourth model (50) can be understood as being the same artificial intelligence inference model, but trained using different training data. Preferably, the third model (40) and the fourth model (50) can be understood as being trained using training data in which only the labeling information on the same X-ray image is labeled differently.

[0207] Preferably, the first model (20) and the second model (30) are the same artificial intelligence inference model, the third model (40) and the fourth model (50) are the same artificial intelligence inference model, and the first model (20) and the third model (40) may be artificial intelligence inference models that process data in different ways.

[0208]

[0209] As described above, the third model (40) and the fourth model (50) are identical artificial intelligence inference models, and their internal structures and data processing methods may be identical. Specifically, if the third model (40) is a CNN-based artificial intelligence inference model, the fourth model (50) may also be a CNN-based artificial intelligence inference model. Conversely, if the third model (40) is a Transformer-based artificial intelligence inference model, the fourth model (50) may also be a Transformer-based artificial intelligence inference model.

[0210]

[0211] As illustrated in FIG. 7b, the third model (40) has a high probability of determining the arthritis grade in the X-ray image as low (compared to the fourth model (50)), and conversely, the fourth model (50) has a high probability of determining the arthritis grade in the X-ray image as high (compared to the third model (40)).

[0212] Specifically, an X-ray image is input into the third model (40) that has completed learning to output third judgment information, and an X-ray image is input into the fourth model (50) that has completed learning to output fourth judgment information, and at this time, there may be a higher probability that the arthritis grade in the fourth judgment information is lower than the arthritis grade in the third judgment information.

[0213] For example, for the same X-ray image in FIG. 7b where the KL grade is level 2, the third model (40) can determine with a certain probability that the X-ray image is level 1, which is lower than level 2.

[0214]

[0215] Preferably, the third model (40) can determine the arthritis grade to be lower than the actual arthritis grade in the X-ray image, but even if the arthritis grade is determined to be lower than the actual arthritis grade, the probability of determining it as the ‘arthritis grade that is lower than the actual arthritis grade and closest (with the smallest difference)’ may be high, or the probability of determining it as the ‘arthritis grade that is lower than the actual arthritis grade and not closest’ or the ‘arthritis grade that is higher than the actual arthritis grade’ may be high.

[0216] For example, when an X-ray image with an arthritis grade of 2 is input into the third model (40), even if the third model (40) determines the arthritis grade to be lower than 2, there is a high probability that it will determine it as ‘1st grade, which is closest to 2nd grade’, or ‘0th grade, which is a greater difference from 2nd grade than 1st grade’, or ‘3rd grade or 4th grade, which is higher than 2nd grade’.

[0217] This is because, as previously mentioned, the third model (40) is trained with training data labeled in the first method, where the value of the arthritis grade corresponding to the ground truth value is 1, and the value of the arthritis grade closest to the arthritis grade corresponding to the ground truth value is determined to be an arbitrary value lower than 1.

[0218] In other words, conceptually, the third model (40) may be an artificial intelligence inference model that optimistically judges the arthritis grade (judging the arthritis grade better than the actual arthritis grade).

[0219]

[0220] Additionally, the fourth model (50) may have a higher probability of determining a higher grade of arthritis in an X-ray image (compared to the third model (40)). Specifically, the grade of arthritis in the fourth judgment information output by the fourth model (50) after training may be higher than the grade of arthritis in the third judgment information output by the third model (40).

[0221] For example, for an X-ray image with a KL grade of 2, the fourth model (50) can determine with a certain probability that the X-ray image is of a higher grade than 2, which is 3.

[0222] Preferably, the fourth model (50) can determine the arthritis grade to be higher than the actual arthritis grade in the X-ray image, but even if the arthritis grade is determined to be higher than the actual arthritis grade, the probability of determining it as ‘the arthritis grade that is higher than the actual arthritis grade and closest (with the smallest difference)’ may be high, while the probability of determining it as ‘the arthritis grade that is higher than the actual arthritis grade and not closest’ or ‘the arthritis grade that is lower than the actual arthritis grade’ may be high.

[0223] For example, when an X-ray image with an arthritis grade of 2 is input to the 4th model (50), even if the 4th model (50) determines the arthritis grade to be higher than 2, there is a high probability that it will determine it as ‘3rd grade, which is closest to 2nd grade’, or ‘4th grade, which is a greater difference from 2nd grade than 3rd grade’, or ‘0th grade or 1st grade, which is lower than 2nd grade’.

[0224] This is because, as previously mentioned, the fourth model (50) is trained with training data labeled in the second method, where the value of the arthritis grade corresponding to the ground truth value is 1, and the value of the arthritis grade closest to the arthritis grade corresponding to the ground truth value is determined to be an arbitrary value lower than 1.

[0225] In other words, conceptually, the fourth model (50) may be an artificial intelligence inference model that judges the arthritis grade pessimistically (judging the arthritis grade worse than the actual arthritis grade).

[0226]

[0227] In this way, the present invention can derive final judgment information regarding the patient's arthritis grade by comprehensively utilizing the third judgment information and the fourth judgment information output by inputting the same X-ray image into the third model (40) and the fourth model (50) that have completed learning.

[0228]

[0229] FIG. 8 illustrates matters related to the information processing method of the third model (40) and the fourth model (50) according to one embodiment of the present invention.

[0230]

[0231] As illustrated in FIG. 8, the first model (20) and the second model (30) are a CNN (Convolutional Neural Network) or a Transformer-based artificial neural network model, and the third model (40) and the fourth model (50) are a Transformer-based artificial neural network model when the first model (20) and the second model (30) are CNN-based artificial neural network models, and a CNN-based artificial neural network model when the first model (20) and the second model (30) are Transformer-based artificial neural network models.

[0232]

[0233] Specifically, the third model (40) and the fourth model (50) may also include a plurality of backbone neural network modules (400, and 500), a plurality of information selection modules (401, and 501), and an integration module (402, and 502), and since the operation of each component is substantially the same as that of FIG. 5, a redundant description will be omitted.

[0234] In addition, the plurality of backbone neural network modules (400, and 500) and the plurality of information selection modules (401, and 501) constituting the third model (40) and the fourth model (50) can process data in the same way. For example, if the third model (40) and the fourth model (50) are CNN-based artificial intelligence inference models, the plurality of backbone neural network blocks (400, and 500) and the plurality of information selection modules (401, and 501) constituting the third model (40) and the fourth model (50) may all be CNN-based artificial intelligence inference models.

[0235] Conversely, when the third model (40) and the fourth model (50) are Transformer-based artificial intelligence inference models, the plurality of backbone neural network blocks (400, and 500) and the plurality of information selection modules (401, and 501) constituting the third model (40) and the fourth model (50) may all be Transformer-based artificial intelligence inference models.

[0236]

[0237] Consequently, the plurality of backbone neural network blocks (200 and 300) and the plurality of information selection modules (201 and 301) constituting the first model (20) and the second model (30) may be artificial neural network blocks that process data in a CNN or Transformer-based manner, and the plurality of backbone neural network blocks (400 and 500) and the plurality of information selection modules (401 and 501) constituting the third model (40) and the fourth model (50) may be artificial neural network blocks that process data in a CNN or Transformer-based manner (in a manner different from the first model (20) and the second model (30).

[0238]

[0239] FIG. 9 illustrates the judgment method of the first model (20) and the third model (40) according to one embodiment of the present invention.

[0240]

[0241] In the following description, the first model (20) and the second model (30) are assumed to be CNN-based artificial intelligence inference models, and the third model (40) and the fourth model (50) are assumed to be Transformer-based artificial intelligence inference models.

[0242]

[0243] In one embodiment of the present invention, a CNN-based artificial intelligence inference model can derive judgment information regarding the grade of arthritis by locally analyzing a specific part of an input X-ray image. Specifically, the CNN-based artificial intelligence inference model operates by dividing the image into small sizes and analyzing the features of each region, thereby analyzing local patterns of specific parts related to arthritis in the X-ray image to derive judgment information.

[0244]

[0245] Conversely, a transformer-based AI inference model can derive judgment information regarding arthritis grade by analyzing the input X-ray image holistically. Specifically, the transformer-based AI inference model operates by analyzing the image holistically using a self-attention algorithm, thereby enabling it to derive judgment information by analyzing the overall patterns of the X-ray image.

[0246]

[0247] FIG. 10 illustrates details regarding a method for determining arthritis grade according to one embodiment of the present invention.

[0248]

[0249] As illustrated in FIG. 10, the first model (20), which is a CNN-based artificial intelligence inference model and trained with training data labeled in the first method, may be a model that makes local judgments from an optimistic perspective regarding X-ray images. In other words, the first model (20) can be understood as representing a doctor who analyzes X-ray images locally from an optimistic perspective in actual medical settings.

[0250] In addition, the second model (30), which is a CNN-based artificial intelligence inference model and trained with training data labeled in the second method, may be a model that makes local judgments from a pessimistic perspective on X-ray images. In other words, the second model (30) can be understood as representing a doctor who analyzes X-ray images locally from a pessimistic perspective in actual medical settings.

[0251] In addition, the third model (40), which is a Transformer-based artificial intelligence inference model and trained with training data labeled in the first method, may be a model that makes an overall judgment from an optimistic perspective regarding X-ray images. In other words, the third model (40) can be understood as representing a doctor who analyzes X-ray images from an optimistic perspective in actual medical settings.

[0252] Additionally, the fourth model (50), which is a transformer-based artificial intelligence inference model and trained with training data labeled in the second method, may be a model that makes an overall judgment from a pessimistic perspective regarding X-ray images. In other words, the fourth model (50) can be understood as representing a doctor who analyzes X-ray images from a pessimistic perspective in actual medical settings.

[0253]

[0254] As such, according to one embodiment of the present invention, the process of determining a patient's arthritis grade by medical staff in an actual medical setting judging X-ray images locally or holistically from a pessimistic or optimistic perspective can be similarly implemented.

[0255]

[0256] FIG. 11 illustrates a pretreatment process according to one embodiment of the present invention.

[0257]

[0258] As illustrated in FIG. 11, an X-ray image can be preprocessed and input into an artificial intelligence inference model (first model (20), second model (30), third model (40), and fourth model (50)), and the preprocessing process may include one or more of resizing, brightness adjustment, histogram extraction, and ORB extraction.

[0259]

[0260] Specifically, the resizing unit (10) can process the size of the X-ray image into a form suitable for an artificial intelligence inference model. For example, the resizing unit (10) can process the X-ray image into a size suitable for a CNN-based artificial intelligence inference model and a Transformer-based artificial intelligence inference model, respectively.

[0261] The brightness control unit (11) can process the brightness of the X-ray image into a form suitable for an artificial intelligence inference model.

[0262] The histogram extraction unit (12) can analyze brightness in an X-ray image and extract a histogram having a brightness distribution of 0 to 255.

[0263] The ORB extraction unit (13) can extract ORB information by recognizing feature points in an X-ray image. Specifically, the ORB extraction unit (13) can extract ORB information for an X-ray image by using the FAST (Features from Accelerated Segment Test) algorithm, which detects local feature points such as edges of an image, and the BRIEF (Binary Robust Independent Elementary Features) algorithm, which converts patches around the recognized feature points into binary vectors.

[0264]

[0265] In this way, X-ray images with adjusted size and brightness, or histograms and ORB information extracted from X-ray images, can be input into an artificial intelligence inference model.

[0266]

[0267] FIG. 12 schematically illustrates the internal configuration of a computing device according to one embodiment of the present invention.

[0268]

[0269] The arthritis grading system illustrated in FIG. 2 described above may include the components of the computing device (11000) illustrated in FIG. 12.

[0270] As illustrated in FIG. 12, the computing device (11000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600). In this case, the computing device (11000) may correspond to the arthritis grading system illustrated in FIG. 2.

[0271] The memory (11200) may include, for example, high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or various other data required for the operation of the computing device (11000).

[0272] At this time, access to memory (11200) from other components such as the processor (11100) or peripheral device interface (11300) can be controlled by the processor (11100).

[0273] The peripheral device interface (11300) can connect input and / or output peripheral devices of the computing device (11000) to the processor (11100) and memory (11200). The processor (11100) can perform various functions for the computing device (11000) and process data by executing software modules or instruction sets stored in the memory (11200).

[0274] The input / output subsystem can connect various input / output peripherals to the peripheral interface (11300). For example, the input / output subsystem may include a controller for connecting peripherals such as a monitor, keyboard, mouse, printer, or, if necessary, a touchscreen or sensor to the peripheral interface (11300). According to another aspect, input / output peripherals may be connected to the peripheral interface (11300) without passing through the input / output subsystem.

[0275] The power circuit (11500) can supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include a power management system, one or more power sources such as a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0276] The communication circuit (11600) can enable communication with another computing device using at least one external port.

[0277] Alternatively, as described above, the communication circuit (11600) may enable communication with other computing devices by including an RF circuit and transmitting and receiving an RF signal, also known as an electromagnetic signal.

[0278] The embodiment of FIG. 12 is merely an example of a computing device (11000), and the computing device (11000) may have some components shown in FIG. 12 omitted, additional components not shown in FIG. 12 added, or a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may include a touchscreen or sensors in addition to the components shown in FIG. 12, and the communication circuit (11600) may include a circuit for RF communication of various communication methods (WiFi, 3G, LTE, Bluetooth, NFC, Zigbee, etc.). The components that can be included in the computing device (11000) may be implemented as hardware, software, or a combination of both hardware and software, including one or more integrated circuits specialized for signal processing or applications.

[0279] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed on a computing device (11000) through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file in response to a request from the computing device (11000).

[0280]

[0281] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0282] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computing devices and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0283] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0284]

[0285] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below are also within the scope of the claims.

Claims

A method for determining arthritis grade performed on a computing system comprising 1.1 or more processors and 1 or more memories, wherein A first judgment information derivation step of preprocessing an X-ray image including a joint region and inputting the preprocessed X-ray image into a first model including an artificial neural network to derive first judgment information regarding the arthritis grade; A second judgment information derivation step of preprocessing an X-ray image including a joint region and inputting the preprocessed X-ray image into a second model including an artificial neural network to derive second judgment information regarding the arthritis grade; and A final judgment step for deriving final judgment information regarding the arthritis grade based on comprehensive judgment information including the first judgment information and the second judgment information; The first model above is a deep learning-based artificial neural network model trained on training data labeled by the first method, and the second model above is a deep learning-based artificial neural network model trained on training data labeled by the second method, and For the same training data, which is an X-ray image, the first method is a method in which the labeling information for the arthritis grade is labeled lower than the second method, and The above arthritis grade is a method for determining arthritis grades that is numerically expressed in proportion to or inversely proportional to the severity of arthritis.

2. In Claim 1, The labeling information for the training data prior to being labeled by the above-mentioned first or second method is, In the stepwise classification of arthritis grades, the arthritis grade corresponding to the ground truth value of the above training data is determined as 1, and the arthritis grade not corresponding to the ground truth value is determined as 0, and is determined as a one-hot vector. The above first method is, In labeling information in the form of a one-hot vector, it is a method of inputting a random number lower than 1 into an arthritis grade lower than the arthritis grade corresponding to 1, and The above second method is, A method for determining arthritis grade, wherein, in the labeling information of the above-mentioned training data, a random number lower than 1 is entered into an arthritis grade higher than the arthritis grade corresponding to 1.

3. In Claim 1, The above-mentioned first model is, It is trained such that the probability of the arthritis grade being determined to be lower than the arthritis grade corresponding to the ground truth value of the above training data occurs, and The above second model is, An arthritis grade determination method that is trained such that the probability of determining an arthritis grade higher than the arthritis grade corresponding to the ground truth value of the above training data occurs.

4. In Claim 1, The above-mentioned first and second models are, It is trained with training data in which labeling information is assigned using either the first method or the second method, with only the labeling method differing, while the same X-ray image is used. The above first judgment information and second judgment information are, A method for determining arthritis grade, comprising information related to numerical values ​​predicted to correspond to each of a plurality of arthritis grades for the input preprocessed X-ray image.

5. In Claim 1, The above-mentioned first model is, A plurality of deep learning-based backbone neural network blocks that receive the above-mentioned preprocessed X-ray image or feature information output from another backbone neural network block of the previous stage, and output feature information at the corresponding stage; A plurality of deep learning-based information screening modules that receive feature information output from the plurality of backbone neural network blocks and output screening information related to the determination of arthritis grade from the corresponding feature information; and An integrated module that receives information including a plurality of screening information output from the plurality of information screening modules and outputs first judgment information regarding the arthritis grade; The above backbone neural network block and information screening module are artificial neural networks that compress data in the same way, a method for determining arthritis grade.

6. In Claim 1, The above arthritis grade determination method is, A third judgment information derivation step for preprocessing an X-ray image including a joint region and inputting the preprocessed X-ray image into a third model including an artificial neural network to derive third judgment information regarding the arthritis grade; The method further includes a fourth judgment information derivation step of preprocessing an X-ray image including a joint region and inputting the preprocessed X-ray image into a fourth model including an artificial neural network to derive fourth judgment information regarding the arthritis grade; The above comprehensive judgment information further includes third judgment information and fourth judgment information, and The above third and fourth models are artificial neural networks that process or compress data in a different way from the above first and second models, a method for determining arthritis grade.

7. In Claim 6, The above first and second models are CNN (Convolutional Neural Network) or Transformer-based artificial neural network models, and The above third and fourth models are, In the case where the above first model and second model are CNN-based artificial neural network models, they are Transformer-based artificial neural network models, and A method for determining arthritis grade, wherein the first and second models are Transformer-based artificial neural network models, and the CNN-based artificial neural network model.

8. In Claim 6, A method for determining arthritis grade, wherein the third model is a deep learning-based artificial neural network model trained on training data labeled by the first method, and the fourth model is a deep learning-based artificial neural network model trained on training data labeled by the second method. An arthritis grade determination system comprising a processor of 9.1 or higher and 1 or more memories, A first judgment information derivation unit that preprocesses an X-ray image including a joint region and inputs the preprocessed X-ray image into a first model including an artificial neural network to derive first judgment information regarding the arthritis grade; A second judgment information derivation unit that preprocesses an X-ray image including a joint region and inputs the preprocessed X-ray image into a second model including an artificial neural network to derive second judgment information regarding the arthritis grade; and A final judgment unit that derives final judgment information regarding the arthritis grade based on comprehensive judgment information including the first judgment information and the second judgment information; The first model above is a deep learning-based artificial neural network model trained on training data labeled by the first method, and the second model above is a deep learning-based artificial neural network model trained on training data labeled by the second method, and For the same training data, which is an X-ray image, the first method is a method in which the labeling information for the arthritis grade is labeled lower than the second method, and The above arthritis grade is an arthritis grade assessment system that is numerically expressed in proportion to or inversely proportional to the severity of arthritis. A computer-readable recording medium comprising 10.1 or more processors and 1 or more memories, and performing an arthritis grade determination method, The above computer-readable recording medium stores instructions for performing the following steps, and The steps below above are, A first judgment information derivation step of preprocessing an X-ray image including a joint region and inputting the preprocessed X-ray image into a first model including an artificial neural network to derive first judgment information regarding the arthritis grade; A second judgment information derivation step of preprocessing an X-ray image including a joint region and inputting the preprocessed X-ray image into a second model including an artificial neural network to derive second judgment information regarding the arthritis grade; and A final judgment step for deriving final judgment information regarding the arthritis grade based on comprehensive judgment information including the first judgment information and the second judgment information; The first model above is a deep learning-based artificial neural network model trained on training data labeled by the first method, and the second model above is a deep learning-based artificial neural network model trained on training data labeled by the second method, and For the same training data, which is an X-ray image, the first method is a method in which the labeling information for the arthritis grade is labeled lower than the second method, and The above arthritis grade is a computer-readable recording medium that is numerically expressed in proportion to or inversely proportion to the severity of arthritis.

Images