Method and apparatus for providing confidence information on result of artificial intelligence model

Abstract

To provide a method and an apparatus for providing confidence information for a result of an artificial intelligence model using at least one influence variable that affects the result of the artificial intelligence model.SOLUTION: A computing apparatus operated by at least one processor includes a target artificial intelligence model that learns at least one task and performs a task for an input medical image to output a target result. The processor includes: obtaining at least one influence variable (impact factor) that affects the target result; and estimating confidence information for the target result on the basis of the influence variable. The influence variable is extracted from additional information of the input medical image, is inferred from the input medical image, is obtained from an external server or a database, or receives an input from a user.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to artificial intelligence-based prediction technology.

Background Art

[0002] Machine-learning technology has provided results that exceed the performance of existing methods in the analysis of various forms of data such as images, voices, and texts. In addition, machine-learning technology has been applied to various fields due to its inherent scalability and flexibility, and various types of neural networks have been published.

[0003] Artificial Intelligence (AI) technology based on machine learning has been actively introduced in the medical field. Previously, a CAD (Computer Aided Detection) device detected lesions based on rules or detected lesions in candidate regions set in medical images, but recent AI-based medical image interpretation technology can analyze the entire image with an AI algorithm and visually provide abnormal lesions.

[0004] Medical staff can receive information on abnormal lesions included in medical images from a diagnostic assistance device that has realized AI-based medical image interpretation technology and make a diagnosis with reference to this information.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present disclosure provides a method and apparatus for providing reliability information for the result of an artificial intelligence model by using at least one influencing variable that affects the result of the artificial intelligence model.

Means for Solving the Problems

[0006] ​A computing device operated by at least one processor, comprising: an objective artificial intelligence model that learns at least one task, performs the task on an input medical image, and outputs an objective result; and a reliability prediction model that obtains at least one impact factor that affects the objective result based on the input medical image, and estimates reliability information of the objective result using the impact factor.

[0007] The impact factor can be determined based on the characteristics of the objective artificial intelligence model and / or the characteristics of the input medical image.

[0008] The impact factor can include at least one of a task-related medical variable of the objective artificial intelligence model, an input image-related variable of the objective artificial intelligence model, a disease-related variable detected by the objective artificial intelligence model, a patient-related demographic variable, and a patient characteristic-related variable.

[0009] The impact factor can be extracted from additional information of the input medical image, inferred from the input medical image, obtained from an external server / database, or receive input from a user.

[0010] The impact factor extracted from the additional information of the input medical image can include at least one of age, gender, and imaging information including the imaging method. The impact factor inferred from the input medical image can include at least one of tissue density, whether an object is included in the input medical image, lesion type, and lesion size change.

[0011] The objective artificial intelligence model can include a model trained to detect lesions from medical images, infer medical diagnosis information or treatment information.

[0012] The reliability prediction model can include a model that learns the relationship between at least one impact factor related to the training medical image and the reliability of the objective result inferred from the training medical image.

[0013] The reliability information of the target result can be provided together with the target result, used for correcting the target result, used as an indicator for recommending re - imaging and / or imaging methods of the input medical image, or used as an indicator for discarding the target result output from the target artificial intelligence model.

[0014] According to one embodiment, there is a method of operating a computing device operated by at least one processor, including the steps of receiving a medical image input into a target artificial intelligence model, obtaining at least one influencing variable that affects the target result output from the target artificial intelligence model based on the medical image, and estimating the reliability information of the target result using the influencing variable.

[0015] The step of obtaining the at least one influencing variable can extract the influencing variable from the additional information of the medical image, infer the influencing variable from the medical image, obtain the influencing variable from an external server / database, or receive an input of the influencing variable from a user.

[0016] The influencing variable extracted from the additional information of the medical image can include at least one of age, gender, and imaging information including the imaging method. The influencing variable inferred from the medical image can include at least one of tissue density, whether an object is included in the medical image, lesion type, and lesion size change.

[0017] When the medical image is a mammogram, the density inferred from the mammogram can be determined as the influencing variable in the step of obtaining the at least one influencing variable. When the medical image is a chest X - ray image, the PA (Posterior Anterior) or AP (Anterior Posterior) information extracted from the additional information of the chest X - ray image can be determined as the influencing variable.

[0018] The operation method may further include the step of correcting the reliability information of the target result using the target result and providing the corrected reliability information as the final reliability information of the target result.

[0019] The operation method may further include the step of providing the reliability information together with the target result.

[0020] The operation method may further include the step of correcting the target result using the reliability information of the target result.

[0021] The operation method may further include the step of discarding the target result when the reliability information for the target result is below a standard.

[0022] The operation method may further include the step of requesting a reshoot of the medical image input to the target artificial intelligence model or recommending a shooting method based on the reliability information for the target result.

[0023] A computing device according to an embodiment, comprising a processor that, when a target result is output from a target artificial intelligence model that has received an input of a medical image, estimates the reliability information of the target result based on at least one impact factor that affects the target result, and provides the target result and the reliability information to a user interface screen.

[0024] The processor may receive the medical image, obtain the impact variable determined based on the characteristics of the target artificial intelligence model and / or the characteristics of the medical image, and then estimate the reliability information of the target result from the impact variable.

[0025] The processor can perform at least one of the following operations: using the target result to correct the reliability information of the target result, and providing the corrected reliability information as the final reliability information of the target result; using the reliability information of the target result to correct the target result, and providing the corrected target result; requesting re - imaging of the medical image or recommending an imaging method based on the reliability information for the target result; and discarding the target result when the reliability information for the target result is below a standard.

Advantages of the Invention

[0026] According to an embodiment, the reliability information for the result of the artificial intelligence model can be provided in a form that can be understood by the user.

[0027] According to an embodiment, the reliability information of the result can be visually provided together with the result of the artificial intelligence model.

Brief Description of the Drawings

[0028]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0029] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those with ordinary knowledge in the technical field to which the present disclosure belongs can easily implement them. However, the present disclosure can be realized in various different forms and is not limited to the embodiments described herein. And in order to clearly explain the present disclosure in the drawings, parts that are unnecessary for explanation are omitted, and similar parts throughout the specification are given similar drawing reference numerals.

[0030] In the description, when a certain part "includes" a certain component, this means that other components can be further included, rather than excluding other components unless there is a contrary description. Also, terms such as "··· part", "··· machine", "module", etc. described in the specification mean a unit that processes at least one function or operation, and this can be realized by hardware, software, or a combination of hardware and software.

[0031] The device of the present disclosure is a computing device configured and connected so that at least one processor can perform the operations of the present disclosure by executing instructions. The computer program includes instructions described so that the processor can execute the operations of the present disclosure and can be stored in a non-transitory computer readable storage medium. The computer program can be downloaded through a network or sold in a product form.

[0032] The medical images of the present disclosure may be images of various parts taken in various modalities. For example, the modalities of medical images may be X-ray, MRI (magnetic resonance imaging), ultrasound, CT (computed tomography), MMG (Mammography), DBT (Digital breast tomosynthesis), etc.

[0033] The user of the present disclosure may be a medical professional such as a doctor, nurse, clinical laboratory technician, sonographer, or medical imaging expert, or may be a member of the general public such as a patient or a caregiver, but is not limited thereto.

[0034] The artificial intelligence model (AI model) of the present disclosure is a machine learning model that learns at least one task and can be realized by a computer program executed by a processor. The artificial intelligence model can be realized by a computer program executed on a computing device, downloaded through a network, or sold in a product form. Alternatively, the artificial intelligence model can be linked with various devices through a network.

[0035] The task that the artificial intelligence model learns can refer to the problem to be solved through machine learning or the operation to be performed through machine learning. For example, when performing recognition, classification, prediction, etc. from a medical image, each of recognition, classification, and prediction can correspond to an individual task.

[0036] In the present disclosure, the result of the artificial intelligence model means the output provided by the artificial intelligence model that has learned the task for the input. The result of the artificial intelligence model can mean various inference / prediction results analyzed from the input, such as an abnormality score, a CAD (Computer Aided Detection) result, a cancer prediction, etc.

[0037] This disclosure provides confidence information for the results of an artificial intelligence model, and the confidence information can include a confidence measure. On the other hand, generally, a confidence score means an indicator showing how accurate the result obtained by an artificial intelligence model for a task is. Therefore, in order to distinguish it from the confidence score of an artificial intelligence model for a task, the confidence information for the results of the artificial intelligence model in this disclosure can be called a confidence measure score.

[0038] FIG. 1 is a diagram for explaining a computing device according to an embodiment, and FIG. 2 is a diagram for explaining an interlocking environment of the computing device according to an embodiment.

[0039] Referring to FIG. 1, a computing device 10 operated by at least one processor provides confidence information for the results of a target artificial intelligence (AI) model 100. Here, the confidence information for the results of the target AI model 100 can be estimated using at least one impact factor that affects the results of the target AI model 100. The results of the target AI model 100 can also be simply called target results. The confidence information for the target results is provided for each target result and can be provided, for example, in terms of lesion units or case units.

[0040] According to a general artificial intelligence model, the confidence in the results can be determined through the likelihood of the results provided by the artificial intelligence model. For example, according to an artificial intelligence model for detecting lesions from medical images, the confidence in the lesion detection results is determined based on the likelihood of the results provided by the artificial intelligence model. However, it is difficult for users to clearly interpret the meaning of the likelihood, and it is difficult to intuitively associate and recognize the likelihood with the confidence in the results.

[0041] Accordingly, the computing device 10 according to various embodiments of the present disclosure can provide the user with reliability information regarding the target result in a form understandable to the user. The computing device 10 can predict, as a target result, a score indicating whether at least a part of the medical image is abnormal, and numerically display the reliability of the target result. For example, the computing device 10 can estimate the abnormality score for the subject to be 71 and estimate the reliability for the abnormality score of 71 to be 98%. The computing device 10 can correct the target result using the reliability information regarding the target result and provide the corrected target result. Alternatively, the computing device 10 can correct the reliability information regarding the target result using the target result and provide the corrected reliability information as the final reliability information. Alternatively, the computing device 10 can use the reliability information regarding the target result to improve the target artificial intelligence model 100.

[0042] When the reliability information regarding the target result is below a reference value, the computing device 10 can discard the target result and recommend to the user that the medical image be re-taken, or recommend a shooting method for obtaining a target result with high reliability information. The computing device 10 can create a report to be provided to the user based on the reliability information regarding the target result. Thus, the reliability information of the target result estimated by the computing device 10 can be provided to the user together with the target result, used for correcting the target result, used as an indicator for recommending to the user re-taking and / or a shooting method of the medical image, or used as an indicator for discarding the target result analyzed by the target artificial intelligence model 100. Among other things, the reliability information regarding the target result can be used as an indicator for improving the performance of the target artificial intelligence model.

[0043] The computing device 10 can include a target artificial intelligence model 100 and a confidence prediction model 200 that provides reliability information regarding the result thereof.

[0044] For the convenience of explanation, the operations are described as being performed on a single computing device 10 below. However, the present disclosure is not limited to such examples, and at least one of the operations described below may be performed on multiple computing devices and / or user terminals depending on the implementation method. For example, each of the target artificial intelligence model 100 and the trust prediction model 200 may be driven by a separate computing device, and the user terminal can receive the output results of the target artificial intelligence model 100 and the trust prediction model 200 and provide them on the screen of the user terminal.

[0045] The target artificial intelligence model 100 is a model that has learned at least one task, and performs the task on the input to output the target result. The target artificial intelligence model 100 learns the relationship between the input and the output during the training process and can output the result inferred from the new input. Here, the input of the target artificial intelligence model 100 may be various types of data, but for the sake of explanation, it is assumed to be a medical image. For example, the target artificial intelligence model 100 may be a model that receives the input of a medical image and is trained to detect lesions from the medical image or infer medical diagnosis information or treatment information. The medical information can include various medical inference / prediction results such as the abnormality score of the lesion and the risk of disease onset predicted based on the lesion.

[0046] The trust prediction model 200 can obtain at least one impact factor that affects the result (target result) of the target artificial intelligence model 100 and provide trust information for the target result based on the impact factor. The impact factor can be determined based on the characteristics of the target artificial intelligence model 100 and its input (e.g., medical image).

[0047] The reliability prediction model 200 can include a model that learns the relationship between at least one influencing variable related to the training medical image and the reliability of the target result inferred from the training medical image. Here, the target result inferred from the training medical image may be the output of the target artificial intelligence model 100. Further, the reliability prediction model 200 can include a model trained to infer at least one influencing variable from the input medical image.

[0048] As an example, the reliability prediction model 200 can obtain various clinical information of a patient in conjunction with a Picture Archiving and Communication System (PACS), an Electronic Medical Record (EMR), and / or an Electronic Health Record (EHR). The reliability prediction model 200 can extract or infer influencing variables from the various clinical information of the patient.

[0049] As another example, the reliability prediction model 200 can extract influencing variables from the additional information (metadata) included in the input image. For example, when the input image conforms to the DICOM (Digital Imaging and Communications in Medicine) standard, the reliability prediction model 200 can extract at least one influencing variable from imaging information including age, gender, imaging method, etc. stored in the header of the DICOM data.

[0050] As still another example, the reliability prediction model 200 can infer influencing variables from the input image, such as tissue density, whether the image contains an object, image quality, type of lesion (e.g., Soft tissue, Calcification, etc.), and size change of the lesion in the image analyzed by artificial intelligence through comparison with past images. For this purpose, the reliability prediction model 200 can include an artificial intelligence model trained to infer at least one influencing variable from the input image.

[0051] In addition, the trust prediction model 200 can obtain influencing variables in conjunction with an external server / database. Alternatively, the trust prediction model 200 can receive input of influencing variables from a user.

[0052] The trust prediction model 200 can estimate trust information of a target result using the influencing variables vectorized in a specified manner. For example, numerical influencing variables can be quantified with values between 0 and 1, and categorical influencing vectors can be quantified by one-hot encoding. For example, based on breast density inferred from an input image, the value of the density influencing variable can be quantified as 1 if it is dense, 0 if not, or with a value between 0 and 1. Alternatively, since the influencing variables obtained from metadata, an external server / database, and user input are relatively accurate values, they can be quantified as 0 or 1, and the influencing variables inferred by the trust prediction model 200 can be quantified with values between 0 and 1 based on the inference probability.

[0053] The trust prediction model 200 can estimate trust information for each result of the target artificial intelligence model 100. Here, the trust information for the result of the target artificial intelligence model 100 can be simply referred to as a confidence measure score.

[0054] The trust prediction model 200 can be implemented by an artificial intelligence model trained to estimate trust information from at least one influencing variable, or a statistical model that calculates trust information from at least one influencing variable. For example, the statistical model can include statistical information regarding the influence that at least one influencing variable has on a target result (e.g., Abnormality score).

[0055] The influencing variables that affect the target results of the artificial intelligence model can be determined from among candidate variables such as task-related medical variables, input image-related variables, disease-related variables, patient-related demographic variables, and patient characteristic-related variables of the target artificial intelligence model 100. For example, when the artificial intelligence model receives an input of a medical image and outputs a target result such as a prediction / inference for this, the reliability prediction model 200 can use at least some of the variables as shown in Table 1 as influencing variables that affect the results of the artificial intelligence model.

[0056]

Table 1

[0057] The reliability prediction model 200 can use task-related medical variables as influencing variables. For example, when the input image is a mammogram, it is known that the higher the density, the more difficult it is for the artificial intelligence model to accurately detect lesions, and the higher the false positive (FP) probability. Thus, since the density related to the reading difficulty is an influencing variable that affects the results, the reliability prediction model 200 can estimate the reliability information for the results of the target artificial intelligence model 100 based on the density of the tissue inferred from the image. At this time, the reliability prediction model 200 can receive the input of the image input to the target artificial intelligence model 100, infer the density of the image, and then use this to estimate the reliability information for the results of the target artificial intelligence model 100.

[0058] The reliability prediction model 200 can use various input image-related variables as influencing variables. For example, the imaging information of the medical image, whether the image contains an object, the image quality, the tissue density, the lesion type, the change in lesion size, etc. can be used as influencing variables. The input image-related variables can be extracted from the additional information of the medical image (for example, the header of DICOM data, the metadata of the medical image, etc.) or inferred from the medical image.

[0059] The shooting information can include the shooting method (e.g., PA / AP), shooting posture, shooting position, shooting equipment, etc. The reliability prediction model 200 can extract the shooting information from the additional information included in the input image or obtain the shooting information from an external server / database. The reliability prediction model 200 can estimate the reliability information for the result of the target artificial intelligence model 100 using the shooting information of the image input to the target artificial intelligence model 100. For example, in the case of a chest X-ray image, it is known that a PA (Posterior Anterior) image is more difficult to interpret than an AP (Anterior Posterior) image. Such difficulty in interpretation is also experienced by the artificial intelligence model. Therefore, when a chest X-ray image is input to the artificial intelligence model, information on whether the chest X-ray image is a PA image or an AP image can be used as an influencing variable. On the other hand, the type and characteristics of the shooting equipment can affect the result of the artificial intelligence model. For example, X-ray devices include the Film method, CR (Computed Radiography) method, DR (Digital Radiography) method, etc., and the shooting method can affect the result of the artificial intelligence model. Also, the manufacturing company and product version of the shooting equipment can affect the result of the artificial intelligence model.

[0060] The objects included in the image can include medical devices, extrinsic materials, buttons, markers, tubes, pacemakers, etc. For example, when an object that is not the patient's biological tissue is included in the image, it is known that the interpretation reliability decreases. Therefore, the reliability prediction model 200 can receive the input of the image input to the target artificial intelligence model 100, extract the objects in the image, and then use this to estimate the reliability information for the result of the target artificial intelligence model 100.

[0061] Image quality can also be used as an influencing variable that affects the results of the target artificial intelligence model. For example, when the quality of the image is poor, such as when the image contains artifacts or has a low resolution, the interpretation confidence is known to decrease. Therefore, the trust prediction model 200 can receive the input of the image input to the target artificial intelligence model 100, obtain the image quality, and then use this to estimate the trust information for the results of the target artificial intelligence model 100.

[0062] Various information inferred from the image can be used as an influencing variable that affects the results of the target artificial intelligence model. As explained in the relationship between breast density and interpretability, the tissue density inferred from the image can be used for estimating the confidence of the target result. Lesion types such as soft tissue / calcification lesions can be inferred from the image, and the lesion type can be used as an influencing variable that affects the results of the target artificial intelligence model. Also, using the patient's past images related to the image to be analyzed, the change in lesion size can be inferred, and the change in lesion size can be used as an influencing variable that affects the results of the target artificial intelligence model.

[0063] Disease-related variables can be used as influencing variables. In the case of an artificial intelligence model for predicting diseases, the detection accuracy may vary depending on the amount of disease-related training data and disease characteristics. Therefore, disease-related variables such as disease type and disease detection accuracy can affect the results of the artificial intelligence model. The trust prediction model 200 can use the disease-related variables detected by the target artificial intelligence model 100 to estimate the trust information for the results of the target artificial intelligence model 100.

[0064] In addition, patient-related demographic variables / patient characteristic-related variables can be used as influencing variables. For example, patient-related demographic variables such as gender, age, race, and regional characteristics can affect the results of the artificial intelligence model. Also, patient characteristic-related variables such as patient personal information can affect the results of the artificial intelligence model. The trust prediction model 200 can extract patient-related demographic variables / patient characteristic-related variables from the additional information included in the input image or obtain them from an external server / database. Or, the trust prediction model 200 can also infer patient-related demographic variables / patient characteristic-related variables from the input image.

[0065] Referring to FIG. 2, the computing device 10 can be constructed as a standalone type or as an interlocking type with other devices. For example, the computing device 10 can be realized to interlock with a plurality of user terminals 20. Also, the computing device 10 can interlock with various databases 30 of a medical institution, such as a Picture Archiving and Communication System (PACS), an Electronic Medical Record (EMR), an Electronic Health Record (EHR), etc., to obtain various clinical information of patients. The user terminal 20 can provide a user interface screen that displays necessary information on the screen in conjunction with the computing device 10 and the database 30. The user terminal 20 can display the information provided from the computing device 10 through a dedicated viewer.

[0066] The computing device 10 may be a server device, and the user terminal 20 may be a client terminal installed in a medical institution, and these can be interlocked through a network. The computing device 10 may be a local server connected to a network within a specific medical institution. The computing device 10 may be a cloud server and can be interlocked with terminals (medical staff terminals) of a number of medical institutions having access rights. The computing device 10 may be a cloud server and can be interlocked with a patient personal terminal having access rights.

[0067] The computing device 10 can receive a request for analysis of a medical image from the user terminal 20 and respond to the user terminal 20 with a target result and reliability information (reliability measurement) for the target result. The computing device 10 can obtain influence variables from the user terminal 20 / database 30. When the medical image is stored in the database 30, the user terminal 20 can transmit the medical image imported from the database 30 to the computing device 10, and the computing device 10 can import the medical image requested by the user terminal 20 from the database 30.

[0068] FIG. 3 is a diagram for explaining a method for providing reliability information for a target result according to another embodiment.

[0069] Referring to FIG. 3, the reliability prediction model 200 can obtain at least one influence variable that affects the result of the target artificial intelligence model 100 and provide reliability information for the target result based on the influence variable.

[0070] The influencing variables of the target artificial intelligence model can be determined from among candidate variables such as task-related medical variables, input image-related variables, disease-related variables, patient-related demographic variables, and patient characteristic-related variables of the target artificial intelligence model 100. For example, the confidence prediction model 200 can extract influencing variables from additional information (metadata) included in the input image. The confidence prediction model 200 can infer the necessary influencing variables from the input image. The confidence prediction model 200 can obtain influencing variables in conjunction with an external server / database. Alternatively, the confidence prediction model 200 can receive input of influencing variables from the user.

[0071] As shown in FIG. 3, the computing device 10 according to one embodiment can provide the initial confidence information output from the confidence prediction model 200 as the confidence information of the target result, but can correct the initial confidence information to provide more accurate final confidence information.

[0072] The computing device 10 can calculate the final confidence information through an adjustment module 300. The computing device 10 can correct the initial confidence information using the target result of the target artificial intelligence model 100. The adjustment module 300 can correct the initial confidence information using the target result, for example, through a weighted sum operation. Alternatively, the adjustment module 300 can be implemented by an artificial intelligence model that has learned the relationship between the input and output, or a statistical model that includes statistical information on the relationship between the input and output.

[0073] On the other hand, according to an embodiment different from the embodiment shown in FIG. 3, the computing device 10 can provide the target result output from the target artificial intelligence model 100, that is, the initial target result, as output information, but can correct the initial target result to provide a more accurate final target result.

[0074] The computing device 10 can calculate the final target result through an adjustment module 300. The computing device 10 can correct the initial target result using the trust information of the trust prediction model 200. For example, the adjustment module 300 can correct the initial target result using the trust information through a weighted sum operation. Alternatively, the adjustment module 300 can be implemented by an artificial intelligence model that has learned the relationship between the input and output, or a statistical model that includes statistical information about the relationship between the input and output.

[0075] FIG. 4 is a flowchart of a method for providing trust information for a target result according to an embodiment.

[0076] Referring to FIG. 4, the computing device 10 receives a medical image input to the target artificial intelligence model 100 (S110). The target artificial intelligence model 100 can receive the input of the medical image from the computing device 10, an external separate device (e.g., a user terminal or a database), or a memory inside the computing device 10.

[0077] The computing device 10 acquires at least one influencing variable that affects the output (target result) of the target artificial intelligence model 100 based on medical images (S120). The computing device 10 can determine the type of influencing variable based on the characteristics of the target artificial intelligence model 100 and / or the characteristics of the input (e.g., medical image). For example, the influencing variable can be determined from among candidate variables such as task-related medical variables (e.g., breast density) of the target artificial intelligence model 100, input image-related variables (e.g., imaging information, whether the image contains an object, image quality, tissue density, lesion type, lesion size change, etc.), disease-related variables (e.g., disease type, disease detection accuracy, etc.), patient-related demographic variables, patient characteristic-related variables, and the like. The computing device 10 can extract at least one influencing variable from the additional information (metadata) included in the input of the target artificial intelligence model 100. The computing device 10 can infer influencing variables (e.g., objects in the image, image quality, tissue density, lesion type, lesion size change, etc.) from the input of the target artificial intelligence model 100. The computing device 10 can acquire the influencing variable in conjunction with an external server / database. Alternatively, the computing device 10 can receive an input of the influencing variable from the user. The computing device 10 can quantify the acquired influencing variable in a specified manner. For example, the computing device 10 can quantify the influencing variable with a value between 0 and 1.

[0078] The computing device 10 estimates the reliability information of the target result output from the target artificial intelligence model 100 using the at least one acquired influencing variable (S130). The computing device 10 can estimate the reliability information for the target result using a reliability prediction model 200 implemented by an artificial intelligence model trained to estimate reliability information from the influencing variable or a statistical model that calculates reliability information from the influencing variable.

[0079] The computing device 10 provides reliability information regarding the target result to the user interface screen (S140). The computing device 10 can provide reliability information regarding the target result to the screen of the user terminal 20. The computing device 10 can provide more accurate final reliability information by correcting the reliability information of the target result using the target result. The computing device 10 can correct the target result using the reliability information of the target result. The computing device 10 can provide the reliability information together with the target result. The reliability information regarding the target result is provided for each target result and can be provided, for example, in units of lesions or cases.

[0080] On the other hand, when the reliability information regarding the target result is below a standard, the computing device 10 can discard the target result. Based on the reliability information regarding the target result, the computing device can request the user for new input in order to obtain a more accurate target result. For example, when the reliability information regarding the target result is below a standard, the computing device can request a reshoot of a medical image or recommend a shooting method for obtaining a target result with high reliability information.

[0081] FIG. 5 is an illustration of an interface screen that provides reliability information regarding a target result of a medical image according to an embodiment.

[0082] Referring to FIG. 5, assume that the target artificial intelligence model 100 outputs an abnormality score of a mammogram image as a target result to the user terminal 20.

[0083] While the target artificial intelligence model 100 infers / predicts the abnormality score of a mammogram, or after inferring / predicting the abnormality score, the computing device 10 can collect at least one influencing variable that affects the image interpretation of the target artificial intelligence model 100. The computing device 10 determines the density of the mammogram as an influencing variable, and can estimate the confidence measure score of the abnormality score output by the target artificial intelligence model 100 using the density obtained from the mammogram. In addition to density, the computing device 10 can use at least one influencing variable determined from candidate variables such as input image-related variables (e.g., imaging information, objects in the image, image quality, lesion type, lesion size change, etc.), disease-related variables (e.g., disease type, disease detection accuracy, etc.), patient-related demographic variables, patient characteristic-related variables, etc. to estimate the confidence information of the abnormality score.

[0084] The user terminal 20 displays the information provided by the computing device 10 on the user interface screen 21. For example, the screen 21 can display nodules detected from the mammogram as a heat map. Then, the screen 21 can display the confidence measure score of the breast cancer suspicion score, together with the breast cancer suspicion score (Abnormality score) that can be displayed from the nodules. For example, the screen 21 can display that the breast cancer suspicion score (probability) is 95%, and the confidence of the result of 95% is 98%.

[0085] The user can directly make a final judgment on whether there is an abnormal lesion and / or subsequent measures, etc., by referring to the analysis results of artificial intelligence. At this time, the computing device 10 according to various embodiments of the present disclosure helps the user to make a final judgment by referring to the analysis results of artificial intelligence by providing the user with the reliability information for the analysis results of artificial intelligence (target results). In addition, the computing device 10 according to various embodiments of the present disclosure can improve the accuracy of the analysis results of artificial intelligence by reflecting the estimated reliability information in the analysis results of artificial intelligence (target results).

[0086] FIG. 6 is a hardware configuration diagram of a computing device according to an embodiment.

[0087] Referring to FIG. 6, the computing device 10 may include one or more processors 11, a memory 13 for loading a computer program executed by the processor 11, a storage 15 for storing the computer program and various data, a communication interface 17, and a bus 19 for connecting them. In addition, the computing device 10 may further include various components. When the computer program is loaded into the memory 13, it may include instructions for the processor 11 to perform the methods / operations according to various embodiments of the present disclosure. That is, the processor 11 can perform the methods / operations according to various embodiments of the present disclosure by executing the instructions. The computer program is composed of a series of computer-readable instructions combined based on functions and indicates what is to be executed by the processor. The computer program may include a target artificial intelligence model trained to infer medical diagnosis information or treatment information from an input image, a reliability prediction model for providing reliability information for the inference results of the target artificial intelligence model, and / or an adjustment model for correcting the initial reliability information and / or the initial target results.

[0088] Processor 11 controls the overall operation of each component of computing device 10. Processor 11 can be configured to include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), or any form of processor well known in the technical field of the present disclosure. Further, processor 11 can perform operations on at least one application or computer program for executing the methods / operations according to various embodiments of the present disclosure.

[0089] Memory 13 stores various data, instructions, and / or information. Memory 13 can load one or more computer programs from storage 15 for executing the methods / operations according to various embodiments of the present disclosure. Memory 13 can be implemented with a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

[0090] Storage 15 can non-temporarily store computer programs. Storage 15 can be configured to include a non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, etc., a hard disk, a removable disk, or any form of computer-readable recording medium well known in the technical field to which the present disclosure pertains.

[0091] Communication interface 17 supports wired and wireless Internet communication of computing device 10. Further, communication interface 17 can also support various communication methods other than Internet communication. For this purpose, communication interface 17 can be configured to include a communication module well known in the technical field of the present disclosure.

[0092] Bus 19 provides an inter-component communication function for computing device 10. Bus 19 can be implemented as various forms of buses, such as an address bus, a data bus, and a control bus. Bus).

[0093] The embodiments of the present disclosure described above are not only realized through devices and methods, but can also be realized through a program that realizes the functions corresponding to the configurations of the embodiments of the present disclosure or a recording medium on which the program is recorded.

[0094] Although the embodiments of the present disclosure have been described in detail above, the scope of rights of the present disclosure is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present disclosure defined in the following claims also belong to the scope of rights of the present disclosure.

Claims

1. A computing device operated by at least one processor, comprising: A target artificial intelligence model that learns at least one task, performs the task on the input medical image, and outputs a target result; The processor obtains at least one impact factor that affects the target result, and estimates reliability information of the target result based on the impact factor; The influencing variables are extracted from additional information of the input medical image, inferred from the input medical image, obtained from an external server or database, or input from a user to a computing device.

2. The computing device of claim 1 , wherein the influence variables extracted from the additional information of the input medical image include at least one of the patient's age, gender, race, regional characteristics, shooting information of the input medical image, whether the input medical image contains an object, image quality, tissue density, lesion type, or lesion size change.

3. 2. The computing device of claim 1, wherein the influencing variables inferred from the input medical image include at least one of: imaging information of the input medical image, whether the input medical image contains an object, image quality, tissue compaction, lesion type, lesion size change, or disease-related variables.

4. The computing device of claim 3 , wherein the shooting information includes at least one of a shooting method, a shooting attitude, a shooting position, or a shooting equipment.

5. The computing device of claim 3 , wherein the image quality includes at least one of: resolution, or the presence or absence of artifacts in an image.

6. The computing device of claim 1 , wherein the influence variables are determined based on at least one of characteristics of the target artificial intelligence model and characteristics of the input medical image.

7. The computing device of claim 1 , wherein the influence variables include at least one of task-related medical variables of the target artificial intelligence model, input image-related variables of the target artificial intelligence model, disease-related variables detected by the target artificial intelligence model, patient-related demographic variables, and patient characteristic-related variables.

8. The computing device of claim 1 , wherein the objective artificial intelligence model comprises a model trained to detect pathologies or infer medical diagnostic or treatment information from medical images.

9. 2. The computing device of claim 1, wherein the confidence information of the desired result is provided along with the desired result, used to correct the desired result, used as an indicator to recommend at least one of recapture and capture methods of the input medical image, or used as an indicator to discard the desired result output from the desired artificial intelligence model.

10. 1. A method of operating a computing device operated by at least one processor, comprising: receiving an input medical image; A step of acquiring at least one influence variable that influences a target result output from a target artificial intelligence model that learns at least one task, performs the task on an input medical image, and outputs a target result; and estimating reliability information of the target result based on the influencing variables; The step of acquiring at least one influence variable includes: The method includes extracting the influence variables from additional information of the input medical image, inferring the influence variables from the input medical image, obtaining the influence variables from an external server or database, or receiving input of the influence variables from a user.

11. The method of claim 10, wherein the influencing variables extracted from the additional information of the input medical image include at least one of the following: patient age, sex, race, regional characteristics, shooting information of the input medical image, whether the input medical image contains an object, image quality, tissue compaction, lesion type, or lesion size change.

12. 11. The method of claim 10, wherein the influential variables inferred from the input medical image include at least one of: imaging information of the input medical image, whether the input medical image contains an object, image quality, tissue compaction, lesion type, or lesion size change.

13. The step of acquiring at least one influencing variable includes: The method of claim 10, further comprising determining the influence variables based on at least one of characteristics of the target artificial intelligence model and characteristics of the input medical image.

14. The method of claim 10, wherein the influence variables include at least one of task-related medical variables of the target artificial intelligence model, input image-related variables of the target artificial intelligence model, disease-related variables detected by the target artificial intelligence model, patient-related demographic variables, and patient characteristic-related variables.

15. The method of claim 10 further comprising correcting the desired result using confidence information of the desired result.

16. The method of claim 10 further comprising providing said confidence information along with said intended result.

17. The method of claim 10 further comprising discarding the desired result based on confidence information for the desired result.

18. The method of claim 10, further comprising: requesting re-imaging of the medical image input to the target artificial intelligence model or recommending an imaging method based on confidence information for the target result.

19. When a target result is output from a target artificial intelligence model that receives an input of a medical image, a processor is provided for estimating reliability information of the target result based on at least one impact factor that affects the target result, and providing the target result and the reliability information on a user interface screen; The influencing variables are extracted from additional information of the input medical image, inferred from the input medical image, obtained from an external server or database, or input from a user to a computing device.

20. 1. A method of operating a computing device operated by at least one processor, comprising: When a target result is output from a target artificial intelligence model that receives an input of a medical image, estimating reliability information of the target result based on at least one impact factor that influences the target result; providing the desired results and the confidence information on a user interface screen; The method of operation, wherein the influencing variables are extracted from additional information of the input medical image, inferred from the input medical image, obtained from an external server or database, or input from a user.

Images