Method and apparatus for providing confidence information for the results of artificial intelligence models

Abstract

A computing device operated by at least one processor includes a target artificial intelligence model that learns at least one task, performs the task on an input medical image and outputs a target result, and a confidence prediction model that obtains at least one impact factor that influences the target result based on the input medical image, and estimates confidence information of the target result using the impact factor.

Description

[Technical field]

[0001] The present invention relates to artificial intelligence-based prediction technology. [Background technology]

[0002] Machine learning technology has provided results that surpass the performance of existing methods in analyzing various types of data, including images, voice, text, etc. In addition, machine learning technology is being applied to various fields due to its inherent scalability and flexibility, and various types of neural networks have been made public.

[0003] Machine learning-based artificial intelligence (AI) technology is being actively introduced into the medical field. Previously, computer-aided detection (CAD) devices detected lesions on a rule-based basis or in candidate areas set in medical images, but recent AI-based medical image interpretation technology can analyze the entire image with AI algorithms and visually display abnormal lesions.

[0004] Medical staff can receive information on abnormal lesions contained in medical images from diagnostic support devices equipped with AI-based medical image interpretation technology and use this information to make diagnoses. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure provides a method and apparatus for providing confidence information for the results of an artificial intelligence model using at least one influencing variable that influences the results of the artificial intelligence model. [Means for solving the problem]

[0006] According to one embodiment, a computing device operated by at least one processor includes a target artificial intelligence model that learns at least one task, performs the task on an input medical image and outputs a target result, and a confidence prediction model that obtains at least one impact factor that influences the target result based on the input medical image, and estimates confidence information of the target result using the impact factor.

[0007] The influencing variables may be determined based on characteristics of the target artificial intelligence model and / or characteristics of the input medical image.

[0008] The influence variables may include at least one of task-related medical variables of the target artificial intelligence model, input image-related variables of the target artificial intelligence model, disease-related variables detected by the target artificial intelligence model, patient-related demographic variables, and patient characteristic-related variables.

[0009] The influence variables may be extracted from additional information of the input medical image, inferred from the input medical image, obtained from an external server / database, or input by a user.

[0010] The influence variables extracted from the additional information of the input medical image can include at least one of age, sex, and imaging information including imaging method. The influence variables inferred from the input medical image can include at least one of tissue compactness, whether the input medical image contains an object, lesion type, and lesion size change.

[0011] The target artificial intelligence model can include a model trained to detect pathology or infer medical diagnostic or treatment information from medical images.

[0012] The confidence prediction model may include a model that has learned a relationship between at least one influencing variable associated with training medical images and a confidence of a target outcome inferred from the training medical images.

[0013] The confidence information of the target result may be provided along with the target result, used to correct the target result, used as an indicator to recommend re-capture and / or capture methods for the input medical image, or used as an indicator to discard the target result output from the target artificial intelligence model.

[0014] According to one embodiment, a method for operating a computing device operated by at least one processor includes a step of receiving a medical image input to a target artificial intelligence model, a step of obtaining at least one influence variable that influences a target result output from the target artificial intelligence model based on the medical image, and a step of estimating reliability information of the target result using the influence variable.

[0015] The step of acquiring at least one influence variable may include extracting the influence variable from additional information of the medical image, inferring the influence variable from the medical image, acquiring the influence variable from an external server / database, or receiving input of the influence variable from a user.

[0016] The influence variables extracted from the additional information of the medical image can include at least one of age, sex, and imaging information including imaging method. The influence variables inferred from the medical image can include at least one of tissue compactness, whether the medical image contains an object, lesion type, and lesion size change.

[0017] In the step of acquiring at least one influence variable, if the medical image is a mammogram, density inferred from the mammogram may be determined as the influence variable, or if the medical image is a chest X-ray image, posterior anterior (PA) or anterior posterior (AP) information extracted from additional information of the chest X-ray image may be determined as the influence variable.

[0018] The operating method may further include correcting reliability information of the target result using the target result, and providing the corrected reliability information as final reliability information of the target result.

[0019] The method of operation may further include providing the confidence information along with the desired result.

[0020] The method may further include correcting the target result using confidence information of the target result.

[0021] The operating method may further include discarding the desired result if the reliability information for the desired result is below a reference level.

[0022] The operating method may further include a step of requesting re-imaging of the medical image input to the target artificial intelligence model or recommending an imaging method based on confidence information for the target result.

[0023] According to one embodiment, a computing device includes a processor that, when a target result is output from a target artificial intelligence model that receives input of a medical image, estimates reliability information of the target result based on at least one impact factor that affects the target result, and provides the target result and the reliability information on a user interface screen.

[0024] After receiving the medical image and obtaining the influence variables determined based on the characteristics of the target artificial intelligence model and / or the characteristics of the medical image, the processor can estimate confidence information of the target result from the influence variables.

[0025] The processor may perform at least one of the following operations: correcting reliability information of the target result using the target result and providing the corrected reliability information as final reliability information of the target result; correcting the target result using the reliability information of the target result and providing the corrected target result; requesting re-capture of the medical image or recommending an imaging method based on the reliability information for the target result; and discarding the target result if the reliability information for the target result is below a standard. Effect of the Invention

[0026] According to the embodiment, it is possible to provide reliability information on the results of an artificial intelligence model in a form that can be understood by a user.

[0027] According to an embodiment, the results of the artificial intelligence model can be visually provided along with confidence information for the results. [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 illustrates a computing device according to one embodiment. [Diagram 2] FIG. 1 illustrates a collaborative environment for computing devices according to one embodiment. [Diagram 3] FIG. 13 is a diagram illustrating a method for providing reliability information on a target result according to another embodiment. [Figure 4] 1 is a flowchart of a method for providing confidence information for a target result according to an embodiment. [Diagram 5] FIG. 13 illustrates an example interface screen for providing confidence information for a medical image outcome according to one embodiment. [Figure 6] FIG. 2 is a hardware configuration diagram of a computing device according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure pertains can easily carry out the present disclosure. However, the present disclosure may be realized in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts that are not necessary for the explanation are omitted in order to clearly explain the present disclosure, and similar parts are designated by similar reference numerals throughout the specification.

[0030] In the description, when a part "includes" a certain element, this means that it can further include other elements and not excluding other elements unless otherwise specified. Furthermore, the terms "... unit", "... machine", "module" and the like described in the specification mean a unit that processes at least one function or operation, which can be realized by hardware, software, or a combination of hardware and software.

[0031] The apparatus of the present disclosure is a computing device configured and coupled to perform the operations of the present disclosure by executing instructions on at least one processor. A computer program includes instructions written to cause a processor to perform the operations of the present disclosure and can be stored on a non-transitory computer readable storage medium. The computer program can be downloaded over a network or sold in the form of a product.

[0032] The medical images of the present disclosure may be images of various parts taken using various modalities, for example, the modalities of the medical images may be X-ray, magnetic resonance imaging (MRI), ultrasound, computed tomography (CT), mammography (MMG), digital breast tomosynthesis (DBT), etc.

[0033] A user of the present disclosure may be, but is not limited to, a medical professional such as a doctor, nurse, clinical laboratory technician, sonographer, or medical imaging professional, or a layperson such as a patient or guardian.

[0034] The artificial intelligence model (AI model) of the present disclosure is a machine learning model that learns at least one task, and can be realized by a computer program executed by a processor. The artificial intelligence model can be realized by a computer program executed by a computing device, and can be downloaded via a network or sold in the form of a product. Alternatively, the artificial intelligence model can be linked to various devices via a network.

[0035] The task that an AI model learns can be the problem to be solved through machine learning or the work to be done through machine learning. For example, when performing recognition, classification, prediction, etc. from medical images, recognition, classification, and prediction can each correspond to individual tasks.

[0036] In the present disclosure, the result of an artificial intelligence model means an output provided by an artificial intelligence model that has learned a task for an input. The result of an artificial intelligence model may mean various inference / prediction results analyzed from an input, such as an abnormality score, a Computer Aided Detection (CAD) result, a cancer prediction, etc.

[0037] The present disclosure provides confidence information for the results of an artificial intelligence model, and the confidence information may include a confidence score. Meanwhile, in general, a confidence score refers to an index indicating how accurate the results of an artificial intelligence model for a task are. Therefore, in order to distinguish it from the confidence score for the task of an artificial intelligence model, the confidence information for the results of an artificial intelligence model of the present disclosure may be called a confidence measure score.

[0038] FIG. 1 is a diagram illustrating a computing device according to an embodiment, and FIG. 2 is a diagram illustrating a collaborative environment of a computing device according to an embodiment.

[0039] Referring to FIG 1, a computing device 10 operated by at least one processor provides confidence information for a result of a target artificial intelligence (AI) model 100. Here, the confidence information for the result of the target AI model 100 can be estimated using at least one impact factor that affects the result of the target AI model 100. The result of the target AI model 100 can also be simply referred to as a target result. The confidence information for the target result is provided for each target result, for example, it can be provided on a lesion-by-lesion or case-by-case basis.

[0040] According to a general artificial intelligence model, the reliability of the result can be determined based on the likelihood of the result provided by the artificial intelligence model. For example, according to an artificial intelligence model for detecting lesions from medical images, the reliability of the lesion detection result is determined based on the likelihood of the result provided by the artificial intelligence model. However, it is difficult for a user to clearly interpret the meaning of the likelihood, and it is difficult for the user to intuitively associate the likelihood with the reliability of the result.

[0041] Therefore, the computing device 10 according to various embodiments of the present disclosure may provide the user with confidence information on the target result in a form that the user can understand. The computing device 10 may predict a score on whether or not at least some tissues in a medical image are abnormal as a target result, and may display a confidence level on the target result in a numerical value. For example, the computing device 10 may estimate an abnormality score on the target object to be 71, and estimate a confidence level on the abnormality score 71 to be 98%. The computing device 10 may correct the target result using the confidence information on the target result and provide the corrected target result. Alternatively, the computing device 10 may correct the confidence information on the target result using the target result, and provide the corrected confidence information as final confidence information. Alternatively, the computing device 10 may use the confidence information on the target result to improve the target artificial intelligence model 100.

[0042] If the reliability information for the target result is below a standard, the computing device 10 may discard the target result and recommend the user to retake the medical image or recommend an imaging method for obtaining a target result with high reliability information. The computing device 10 may create a report to be provided to the user based on the reliability information for the target result. In this manner, the reliability information for the target result estimated by the computing device 10 may be provided to the user together with the target result, used to correct the target result, used as an index for recommending retaking and / or an imaging method for the medical image to the user, or used as an index for discarding the target result analyzed by the target artificial intelligence model 100. In addition, the reliability information for the target result may be used as an index for improving the performance of the target artificial intelligence model.

[0043] The computing device 10 can include a target artificial intelligence model 100 and a Confidence Prediction Model 200 that provides confidence information for the results.

[0044] For convenience of explanation, the following description will be made on one computing device 10. However, the present disclosure is not limited to this example, and at least one of the operations described below may be performed on multiple computing devices and / or user terminals depending on the implementation method. For example, the target AI model 100 and the trust prediction model 200 may each be run on a separate computing device, and the user terminal may receive the output results of the target AI model 100 and the trust prediction model 200 and provide them on the screen of the user terminal.

[0045] The target AI model 100 is a model that has learned at least one task, and performs a task for an input to output a target result. The target AI model 100 can learn the relationship between the input and the output during the training process and output a result inferred from a new input. Here, the input of the target AI model 100 may be various data, but in the description, it is assumed to be a medical image. For example, the target AI model 100 may be a model trained to receive an input of a medical image, detect a lesion from the medical image, or infer medical diagnosis information or treatment information. The medical information may include various medical inference / prediction results such as an abnormality score of a lesion, a predicted risk of developing a disease based on the lesion, etc.

[0046] The confidence prediction model 200 can obtain at least one impact factor that affects the result (target result) of the target AI model 100, and provide confidence information for the target result based on the impact factor. The impact factor can be determined based on the characteristics of the target AI model 100 and its input (e.g., medical images).

[0047] The confidence prediction model 200 may include a model that learns a relationship between at least one impact variable associated with a training medical image and a confidence of a target result inferred from the training medical image, where the target result inferred from the training medical image may be an output of the target artificial intelligence model 100. The confidence prediction model 200 may also include a model trained to infer at least one impact variable from an input medical image.

[0048] For example, the trust prediction model 200 may obtain various clinical information of a patient in cooperation with a Picture Archiving and Communication System (PACS), an Electronic Medical Record (EMR), and / or an Electronic Health Record (EHR), etc. The trust prediction model 200 may extract or infer influential variables from the various clinical information of the patient.

[0049] As another example, the confidence prediction model 200 can extract an influence variable from additional information (metadata) included in the input image. For example, if the input image complies with the Digital Imaging and Communications in Medicine (DICOM) standard, the confidence prediction model 200 can extract at least one influence variable from imaging information including age, sex, imaging method, etc. stored in the header of the DICOM data.

[0050] As another example, the confidence prediction model 200 can infer influential variables from the input image, such as tissue compactness, whether the image contains an object, image quality, type of lesion (e.g., soft tissue, calcification, etc.), size change of lesion in the image analyzed by artificial intelligence through comparison with previous images, etc. To this end, the confidence prediction model 200 can include an artificial intelligence model trained to infer at least one influential variable from the input image.

[0051] In addition, the trust prediction model 200 can acquire influence variables by linking with an external server / database, or can receive influence variables from a user.

[0052] The confidence prediction model 200 can estimate confidence information of a target result using influence variables vectorized in a specified manner. For example, a numeric influence variable can be quantified with a value between 0 and 1, and a categorical influence vector can be quantified with one-hot encoding. For example, based on breast compactness inferred from an input image, the value of the compactness influence variable can be quantified with 1 if compact and 0 if not, or can be quantified with a value between 0 and 1. Alternatively, since the influence variables acquired from metadata, an external server / database, and user input are relatively accurate values, they can be quantified with 0 or 1, and the influence variables inferred by the confidence prediction model 200 can be quantified with a value between 0 and 1 based on the inference probability.

[0053] The confidence prediction model 200 can estimate confidence information for each result of the target AI model 100. Here, the confidence information for the result of the target AI model 100 can be simply called a confidence measure score.

[0054] The confidence prediction model 200 can be implemented as an artificial intelligence model trained to estimate confidence information from at least one influencing variable, or a statistical model that calculates confidence information from at least one influencing variable. For example, the statistical model can include statistical information regarding the impact of at least one influencing variable on a target outcome (e.g., abnormality score).

[0055] The influencing variables that affect the target result of the target AI model can be determined from among candidate variables such as task-related medical variables, input image-related variables, disease-related variables, patient-related demographic variables, and patient characteristic-related variables of the target AI model 100. For example, when the AI ​​model receives input of a medical image and outputs a target result such as prediction / inference thereon, the trust prediction model 200 can use at least some of the variables in Table 1 as influencing variables that affect the result of the AI ​​model.

[0056] [Table 1]

[0057] The reliability prediction model 200 can use task-related medical variables as influencing variables. For example, when the input image is a mammography image, it is known that the higher the density, the more difficult it is for the AI ​​model to accurately detect lesions, and the higher the probability of false positives (FP). As such, since density, which is related to the difficulty of interpretation, is an influencing variable that affects the result, the reliability prediction model 200 can estimate reliability information for the result of the target AI model 100 based on the density of tissue inferred from the image. In this case, the reliability prediction model 200 can receive an image input to the target AI model 100, infer the density of the image, and then use the inferred density to estimate reliability information for the result of the target AI model 100.

[0058] The confidence prediction model 200 can use various input image-related variables as influencing variables, such as the photographing information of the medical image, whether the image contains an object, image quality, tissue density, lesion type, lesion size change, etc. The input image-related variables can be extracted from additional information of the medical image (e.g., the header of the DICOM data, metadata of the medical image, etc.) or can be inferred from the medical image.

[0059] The imaging information may include an imaging method (e.g., PA / AP), imaging posture, imaging position, imaging equipment, etc. The reliability prediction model 200 may extract imaging information from additional information included in the input image, or may obtain imaging information from an external server / database. The reliability prediction model 200 may estimate reliability information for the result of the target artificial intelligence model 100 using imaging information of an image input to the target artificial intelligence model 100. For example, in the case of chest X-ray images, it is known that a posterior anterior (PA) image is more difficult to read than an anterior posterior (AP) image. The artificial intelligence model also experiences such difficulty in reading. Therefore, when a chest X-ray image is input to the artificial intelligence model, information on whether the chest X-ray image is a PA image or an AP image can be used as an influencing variable. Meanwhile, the type and characteristics of imaging equipment may affect the result of the artificial intelligence model. For example, X-ray devices include a film type, a computed radiography (CR) type, a digital radiography (DR) type, etc., and the imaging type may affect the result of the artificial intelligence model. In addition, the manufacturer and product version of the photography equipment can affect the results of the artificial intelligence model.

[0060] The objects included in the image may include medical devices, extrinsic materials, buttons, markers, tubes, pacemakers, etc. For example, it is known that when an object other than the patient's biological tissue is included in an image, the reliability of interpretation decreases. Therefore, the reliability prediction model 200 receives the image input to the target AI model 100, extracts the objects in the image, and then uses the extracted objects to estimate reliability information for the results of the target AI model 100.

[0061] Image quality can also be used as an influencing variable that affects the results of the target artificial intelligence model. For example, it is known that poor image quality, such as the inclusion of artifacts in the image or the image having low resolution, reduces the interpretation reliability. Therefore, the confidence prediction model 200 receives the image input to the target artificial intelligence model 100, acquires the image quality, and then uses the image quality to estimate confidence information for the results of the target artificial intelligence model 100.

[0062] Various information inferred from images can be used as influencing variables that affect the results of the target AI model. As explained in the relationship between breast compactness and interpretation difficulty, tissue compactness inferred from images can be used to estimate the reliability of the target results. Lesion types such as soft tissue / calcification lesions can be inferred from images, and the lesion type can be used as an influencing variable that affects the results of the target AI model. In addition, changes in lesion size can be inferred using past images of patients related to the images being analyzed, and the changes in lesion size can be used as an influencing variable that affects the results of the target AI model.

[0063] Disease-related variables can be used as influencing variables. In the case of an AI model predicting a disease, the detection accuracy may vary depending on the amount of disease-related training data and the disease characteristics. Therefore, disease-related variables such as disease type and disease detection accuracy can affect the results of the AI ​​model. The trust prediction model 200 can estimate trust information for the results of the target AI model 100 using the disease-related variables detected by the target AI model 100.

[0064] In addition, patient-related demographic variables / patient characteristic-related variables can be used as influencing variables. For example, patient-related demographic variables such as gender, age, race, and regional characteristics can affect the results of the artificial intelligence model. In addition, patient characteristic-related variables such as patient personal information can affect the results of the artificial intelligence model. The trust prediction model 200 can extract the patient-related demographic variables / patient characteristic-related variables from additional information included in the input image or obtain them from an external server / database. Alternatively, the trust prediction model 200 can infer the patient-related demographic variables / patient characteristic-related variables from the input image.

[0065] Referring to FIG. 2, the computing device 10 may be constructed as a standalone device or in conjunction with other devices. For example, the computing device 10 may be realized to be in conjunction with a plurality of user terminals 20. The computing device 10 may also be in conjunction with various databases 30 of a medical institution, such as a Picture Archiving and Communication System (PACS), an Electronic Medical Record (EMR), an Electronic Health Record (EHR), etc., to acquire various clinical information of a patient. The user terminal 20 may provide a user interface screen that displays necessary information on a screen in conjunction with the computing device 10 and the database 30. The user terminal 20 may display information provided by the computing device 10 through a dedicated viewer.

[0066] The computing device 10 may be a server device, and the user terminal 20 may be a client terminal installed in a medical institution, and these may be linked through a network. The computing device 10 may be a local server connected to a network within a specific medical institution. The computing device 10 may be a cloud server, and may be linked to multiple medical institution terminals (medical staff terminals) that have access authority. The computing device 10 may be a cloud server, and may be linked to a patient's personal terminal that has access authority.

[0067] The computing device 10 can receive a request for analysis of a medical image from the user terminal 20, and respond to the user terminal 20 with a target result and reliability information (reliability measurement degree) for the target result. The computing device 10 can acquire influence variables from the user terminal 20 / database 30. When a medical image is stored in the database 30, the user terminal 20 can transmit the medical image imported from the database 30 to the computing device 10, and the computing device 10 can import the medical image requested by the user terminal 20 from the database 30.

[0068] FIG. 3 is a diagram illustrating a method for providing reliability information on a target result according to another embodiment.

[0069] Referring to FIG. 3, the confidence prediction model 200 can obtain at least one influence variable that influences the result of the target artificial intelligence model 100, and provide confidence information for the target result based on the influence variable.

[0070] The influence variables of the target AI model can be determined from among candidate variables such as task-related medical variables, input image-related variables, disease-related variables, patient-related demographic variables, and patient characteristic-related variables of the target AI model 100. For example, the trust prediction model 200 can extract the influence variables from additional information (metadata) included in the input image. The trust prediction model 200 can infer the necessary influence variables from the input image. The trust prediction model 200 can acquire the influence variables in conjunction with an external server / database. Alternatively, the trust prediction model 200 can receive the input of the influence variables from a user.

[0071] As shown in FIG. 3, a computing device 10 according to one embodiment can provide initial confidence information output from a confidence prediction model 200 as confidence information of a target result, but can also correct the initial confidence information to provide more accurate final confidence information.

[0072] The computing device 10 may calculate the final trust information through an adjustment module 300. The computing device 10 may correct the initial trust information using the target result of the target artificial intelligence model 100. The adjustment module 300 may correct the initial trust information using the target result, for example, through a weighted sum operation. Alternatively, the adjustment module 300 may be realized by an artificial intelligence model that has learned the relationship between inputs and outputs, or a statistical model that includes statistical information on the relationship between inputs and outputs.

[0073] Meanwhile, according to an embodiment different from the embodiment shown in FIG. 3, the computing device 10 can provide the target result output from the target artificial intelligence model 100, i.e., the initial target result, as output information, but can correct the initial target result to provide a more accurate final target result.

[0074] The computing device 10 may calculate a final target result through an adjustment module 300. The computing device 10 may correct an initial target result using trust information of the trust prediction model 200. The adjustment module 300 may correct the initial target result using trust information, for example, through a weighted sum operation. Alternatively, the adjustment module 300 may be realized by an artificial intelligence model that has learned the relationship between inputs and outputs, or a statistical model that includes statistical information on the relationship between inputs and outputs.

[0075] FIG. 4 is a flow chart of a method for providing confidence information for a target result according to one embodiment.

[0076] 4, the computing device 10 receives a medical image input to the target artificial intelligence model 100 (S110). The target artificial intelligence model 100 may receive the medical image input from the computing device 10, an external separate device (e.g., a user terminal or a database), or a memory within the computing device 10.

[0077] The computing device 10 obtains at least one influence variable that influences the output (target result) of the target artificial intelligence model 100 based on the medical image (S120). The computing device 10 can determine the type of influence variable based on the characteristics of the target artificial intelligence model 100 and / or the characteristics of the input (e.g., medical image). For example, the influence variable can be determined from candidate variables such as task-related medical variables (e.g., breast compactness) of the target artificial intelligence model 100, input image-related variables (e.g., shooting information, whether an object is included in the image, image quality, tissue compactness, lesion type, lesion size change, etc.), disease-related variables (e.g., disease type, disease detection accuracy, etc.), patient-related demographic variables, patient characteristic-related variables, etc. The computing device 10 can extract at least one influence variable from additional information (metadata) included in the input of the target artificial intelligence model 100. The computing device 10 can infer the influence variables (e.g., objects in the image, image quality, tissue compactness, lesion type, lesion size change, etc.) from the input of the target artificial intelligence model 100. The computing device 10 may acquire the influence variables by interfacing with an external server / database. Alternatively, the computing device 10 may receive the influence variables from a user. The computing device 10 may convert the acquired influence variables into numerical values ​​in a specified manner. For example, the computing device 10 may convert the influence variables into numerical values ​​between 0 and 1.

[0078] The computing device 10 estimates trust information of the target result output from the target artificial intelligence model 100 using the acquired at least one influence variable (S130). The computing device 10 can estimate trust information for the target result using an artificial intelligence model trained to estimate trust information from influence variables, or a trust prediction model 200 implemented as a statistical model that calculates trust information from influence variables.

[0079] The computing device 10 provides the confidence information for the target result on a user interface screen (S140). The computing device 10 may provide the confidence information for the target result on a screen of the user terminal 20. The computing device 10 may provide more accurate final confidence information by correcting the confidence information of the target result using the target result. The computing device 10 may correct the target result using the confidence information of the target result. The computing device 10 may provide the confidence information together with the target result. The confidence information for the target result is provided for each target result, for example, may be provided on a lesion-by-lesion or case-by-case basis.

[0080] Meanwhile, the computing device 10 may discard the desired result if the reliability information for the desired result is below a standard. The computing device may request a new input from the user to obtain a more accurate desired result based on the reliability information for the desired result. For example, if the reliability information for the desired result is below a standard, the computing device may request a retake of the medical image or recommend an imaging method for obtaining a desired result with high reliability information.

[0081] FIG. 5 is an illustration of an interface screen for providing confidence information for a desired outcome of a medical image according to one embodiment.

[0082] Referring to FIG. 5, it is assumed that the target artificial intelligence model 100 outputs an abnormality score of a mammogram to the user terminal 20 as a target result.

[0083] While the target AI model 100 infers / predicts the abnormality score of the mammogram image or after inferring / predicting the abnormality score, the computing device 10 can collect at least one influencing variable that influences the image interpretation of the target AI model 100. The computing device 10 can determine the density of the mammogram image as an influencing variable and estimate a confidence measure score of the abnormality score output by the target AI model 100 using the density obtained from the mammogram image. In addition to the density, the computing device 10 can estimate the confidence measure score of the abnormality score using at least one influencing variable determined from candidate variables such as input image-related variables (e.g., shooting information, objects in the image, image quality, lesion type, lesion size change, etc.), disease-related variables (e.g., disease type, disease detection accuracy, etc.), patient-related demographic variables, and patient characteristic-related variables.

[0084] The user terminal 20 displays the information provided by the computing device 10 on a user interface screen 21. For example, the screen 21 may display nodules detected from a mammogram in a heat map. The screen 21 may then display a breast cancer suspicion score, which may be displayed as an abnormality score for the nodule, as well as a confidence measure score for the breast cancer suspicion score. For example, the screen 21 may display that the breast cancer suspicion score (probability) is 95%, and the confidence measure score for a result of 95% is 98%.

[0085] The user can directly make a final decision on whether there is an abnormality and / or on follow-up measures by referring to the AI ​​analysis result. At this time, the computing device 10 according to various embodiments of the present disclosure can help the user to make a final decision by referring to the AI ​​analysis result by providing the user with reliability information on the AI ​​analysis result (target result). In addition, the computing device 10 according to various embodiments of the present disclosure can improve the accuracy of the AI ​​analysis result by reflecting the estimated reliability information on the AI ​​analysis result (target result).

[0086] FIG. 6 is a hardware configuration diagram of a computing device according to one embodiment.

[0087] Referring to FIG. 6, the computing device 10 may include one or more processors 11, a memory 13 for loading a computer program executed by the processor 11, a storage 15 for storing the computer program and various data, a communication interface 17, and a bus 19 connecting them. In addition, the computing device 10 may further include various components. The computer program may include instructions for causing the processor 11 to perform the method / operation according to various embodiments of the present disclosure when loaded into the memory 13. That is, the processor 11 may perform the method / operation according to various embodiments of the present disclosure by executing the instructions. The computer program is composed of a series of computer-readable instructions linked based on functions, and indicates something that is executed by the processor. The computer program may include a target artificial intelligence model trained to infer medical diagnosis information or treatment information from an input image, a confidence prediction model providing confidence information for the inference result of the target artificial intelligence model, and / or an adjustment model for correcting the initial confidence information and / or the initial target result.

[0088] The processor 11 controls the overall operation of each component of the computing device 10. The processor 11 may include at least one of a central processing unit (CPU), a microprocessor unit (MPU), a microcontroller unit (MCU), a graphic processing unit (GPU), or any other type of processor well known in the art of the present disclosure. The processor 11 may also perform calculations for at least one application or computer program for executing methods / operations according to various embodiments of the present disclosure.

[0089] The memory 13 stores various data, instructions and / or information. The memory 13 can load one or more computer programs from the storage 15 to execute the methods / operations according to various embodiments of the present disclosure. The memory 13 can be implemented with a volatile memory such as a RAM, although the scope of the present disclosure is not limited in this respect.

[0090] The storage 15 can non-temporarily store a computer program. The storage 15 can include a non-volatile memory such as a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, or any other computer-readable recording medium well known in the technical field to which the present disclosure pertains.

[0091] The communication interface 17 supports wired and wireless Internet communication for the computing device 10. The communication interface 17 may also support various communication methods other than Internet communication. To this end, the communication interface 17 may be configured to include a communication module well known in the art of the present disclosure.

[0092] The bus 19 provides a communication function between the components of the computing device 10. The bus 19 can be realized by various types of buses such as an address bus, a data bus, and a control bus.

[0093] The embodiments of the present disclosure described above can be realized not only through an apparatus and a method, but also through a program that realizes functions corresponding to the configurations of the embodiments of the present disclosure or a recording medium on which the program is recorded.

[0094] Although the embodiments of the present disclosure have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present disclosure defined in the following claims also fall within the scope of the present disclosure.

Claims

1. A computing device operated by at least one processor, comprising: a target artificial intelligence model that learns at least one task and performs the task on an input medical image to output a target result; and A reliability prediction model is provided for obtaining at least one impact factor that affects the target result based on the input medical image, and estimating reliability information of the target result using the impact factor; The influence variables are extracted from additional information of the input medical image, inferred from the input medical image, obtained from an external server / database, or input by a user; The influencing variables extracted from the additional information of the input medical image include at least one of age, sex, and imaging information including an imaging method; The influential variables inferred from the input medical image include at least one of tissue compactness, whether the input medical image contains an object, lesion type, and lesion size change.

2. The computing device of claim 1 , wherein the influence variables are determined based on characteristics of the target artificial intelligence model and / or characteristics of the input medical image.

3. The computing device of claim 1 , wherein the influence variables include at least one of task-related medical variables of the target artificial intelligence model, input image-related variables of the target artificial intelligence model, disease-related variables detected by the target artificial intelligence model, patient-related demographic variables, and patient characteristic-related variables.

4. The computing device of claim 1 , wherein the objective artificial intelligence model comprises a model trained to detect pathologies or infer medical diagnostic or treatment information from medical images.

5. 2. The computing device of claim 1, wherein the confidence prediction model comprises a model that learns a relationship between at least one influence variable associated with training medical images and a confidence of a target result inferred from the training medical images.

6. The computing device of claim 1 , wherein the confidence information of the desired result is provided along with the desired result, used to correct the desired result, used as an indicator to recommend recapture and / or capture methods for the input medical image, or used as an indicator to discard the desired result output from the desired artificial intelligence model.

7. A computing device operated by at least one processor, comprising: a target artificial intelligence model that learns at least one task and performs the task on an input medical image to output a target result; and A reliability prediction model is provided for obtaining at least one impact factor that affects the target result based on the input medical image, and estimating reliability information of the target result using the impact factor; If the input medical image is a mammogram, at least one of the influencing variables is a density inferred from the mammogram, A computing device, wherein when the input medical image is a chest X-ray image, at least one of the influencing variables is posterior anterior (PA) or anterior posterior (AP) information extracted from additional information of the chest X-ray image.

8. 1. A method of operating a computing device operated by at least one processor, comprising: receiving a medical image input to a target artificial intelligence model; obtaining at least one influencing variable that influences a target result output from the target artificial intelligence model based on the medical image; and estimating confidence information of the target result using the influence variables; The step of acquiring at least one influence variable may include extracting the influence variable from additional information of the medical image, inferring the influence variable from the medical image, acquiring the influence variable from an external server / database, or receiving an input of the influence variable from a user; The influence variables extracted from the additional information of the medical image are: At least one of the following information is included: age, sex, and shooting information including shooting method; The influential variables inferred from the medical images are: The method of operation includes at least one of tissue compactness, whether the medical image contains an object, lesion type, and lesion size change.

9. A method of operating a computing device operated by at least one processor, comprising: receiving a medical image input to a target artificial intelligence model; obtaining at least one influencing variable that influences a target result output from the target artificial intelligence model based on the medical image; and estimating confidence information of the target result using the influence variables; The step of acquiring at least one influence variable includes: if the medical image is a mammogram, determining the influencing variable as a density inferred from the mammogram; When the medical image is a chest X-ray image, the operation method comprises determining, as the influencing variable, posterior anterior (PA) or anterior posterior (AP) information extracted from additional information of the chest X-ray image.

10. The method of claim 8 or 9, further comprising the step of correcting confidence information of the target result using the target result and providing the corrected confidence information as final confidence information of the target result.

11. A method according to claim 8 or 9, further comprising the step of providing said confidence information together with said intended result.

12. A method according to claim 8 or 9, further comprising correcting the target result using confidence information of the target result.

13. A method according to claim 8 or 9, further comprising the step of discarding the desired result if the confidence information for the desired result is below a threshold.

14. The method of claim 8 or 9, further comprising the step of requesting re-imaging of the medical image input to the target artificial intelligence model or recommending an imaging method based on confidence information for the target result.

15. When a target result is output from a target artificial intelligence model that receives an input of a medical image, a processor is provided for estimating reliability information of the target result based on at least one impact factor that affects the target result, and providing the target result and the reliability information on a user interface screen; The influencing variables are extracted from additional information of the medical image, inferred from the medical image, obtained from an external server / database, or input by a user; The influencing variables extracted from the additional information of the medical image include at least one of age, sex, and imaging information including an imaging method; The impact variables inferred from the medical image include at least one of tissue compactness, whether the medical image contains an object, lesion type, and lesion size change.

16. A processor for estimating reliability information of a target result based on at least one impact factor that affects the target result when the target result is output from a target artificial intelligence model that receives a medical image as input, and providing the target result and the reliability information on a user interface screen, If the medical image is a mammogram, at least one of the influencing variables is a density inferred from the mammogram, A computing device, wherein when the medical image is a chest X-ray image, at least one of the influencing variables is posterior anterior (PA) or anterior posterior (AP) information extracted from additional information of the chest X-ray image.

17. The computing device of claim 15 or 16, wherein the processor receives the medical image, obtains the influence variables determined based on characteristics of the target artificial intelligence model and / or characteristics of the medical image, and then estimates confidence information of the target result from the influence variables.

18. the processor correcting confidence information of the target result using the target result and providing the corrected confidence information as final confidence information of the target result; correcting the desired result using confidence information of the desired result and providing a corrected desired result; requesting re-imaging of the medical image or recommending an imaging method based on the confidence information for the target result; and 17. A computing device according to claim 15 or 16, further comprising at least one of the following actions: discarding the desired result if confidence information for the desired result is below a threshold.

Images