Method, program and device for building a medical artificial intelligence model

Abstract

According to an embodiment of the present disclosure, a method, a program, and an apparatus for constructing a medical AI model, which are executed by a computing device, are disclosed. The method includes the steps of: establishing an evaluation criterion for an AI model based on a task intended by a user; determining a first index used in a loss calculation for training the AI ​​model and a second index used in an evaluation calculation for selecting the trained model according to the established evaluation criterion; and constructing an AI model that performs the task intended by the user based on the determined first and second indices.

Description

[Technical Field]

[0001] The present disclosure relates to deep learning technology in the medical field, and more particularly to a method for building a medical artificial intelligence model that meets the user's needs. [Background technology]

[0002] To use an AI model, it is important to train the AI ​​model well using high-quality data that matches the domain of use. However, training an AI model using high-quality data does not necessarily mean that the AI ​​model will perform in line with the user's objectives and domain of use. Therefore, it is important to properly evaluate whether the trained model delivers good performance. In other words, evaluation of a trained model can ensure the reliability, efficiency, transparency, etc. of the model and can encourage continuous improvement of the model to suit the user's objectives and domain of use.

[0003] The criteria for evaluating a learning model are generally set independently of the model's learning conditions and parameters. That is, in the process of building an AI model, evaluation is performed on a model that has completed learning, so the evaluation criteria themselves do not directly affect the determination of the model's learning conditions and parameters. Furthermore, the criteria for evaluating a learning model can include not only standard performance indicators but also evaluation indicators specific to the intended use or domain, making it difficult to set the model's learning conditions and parameters to reflect such rapidly changing criteria.

[0004] However, the more an AI model performs in accordance with the evaluation criteria, the better a model can be developed that achieves the ultimate task to be realized through the AI ​​model. Therefore, when it is necessary to include evaluation indicators specific to the purpose of use or domain, such as in the medical field, it is considered necessary to appropriately reflect the evaluation indicators specific to the purpose of use or domain in the entire process of developing a model. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure aims to provide a method for constructing a medical artificial intelligence model by deriving evaluation criteria that define an artificial intelligence model based on the task intended by a user, and configuring parameters for model learning and selection to meet the evaluation criteria.

[0006] However, the problems to be solved by the present disclosure are not limited to those mentioned above, and other problems not mentioned will be clearly understood from the description below. [Means for solving the problem]

[0007] To achieve the above object, one embodiment of the present disclosure discloses a method for constructing a medical AI model performed by a computing device, the method including the steps of: establishing an evaluation criterion for an AI model based on a task intended by a user; determining a first index used in a loss calculation for training the AI ​​model and a second index used in an evaluation calculation for selecting the trained model according to the established evaluation criterion; and constructing an AI model that performs the task intended by the user based on the determined first and second indexes.

[0008] Alternatively, the evaluation criteria may include at least one of a first criterion for the accuracy of the artificial intelligence model, a second criterion for the uncertainty of the output of the artificial intelligence model, or a third criterion for the correlation between the output of the artificial intelligence model and biometric measurements that determine the presence or absence of a disease included in the task intended by the user.

[0009] Alternatively, if the user's intended task is to predict left ventricular systolic dysfunction (LVSD), the biometric measure determining the presence or absence of the disease may be left ventricular ejection fraction (EF).

[0010] Alternatively, the step of establishing evaluation criteria for an artificial intelligence model based on a task intended by a user may include a step of determining the proportions of the first criterion, the second criterion, and the third criterion in the evaluation criteria based on the task intended by the user.

[0011] Alternatively, if the user's intended task is to predict left ventricular systolic dysfunction (LVSD), the proportion of the first criterion, the second criterion, and the third criterion in the evaluation criteria can be determined to be 4:3:3.

[0012] Alternatively, the step of determining a first index used in a loss calculation for training an artificial intelligence model and a second index used in an evaluation calculation for selecting the trained model according to the established evaluation criterion may include a step of determining a loss function included in the first index so that a correlation according to the third criterion can be calculated.

[0013] Alternatively, if the user's intended task is prediction of left ventricular systolic dysfunction (LVSD), the loss function included in the first index may include a left ventricular ejection fraction (EF) regression loss function.

[0014] Alternatively, the step of determining a first index used in a loss calculation for training an artificial intelligence model and a second index used in an evaluation calculation for selecting the trained model based on the established evaluation criteria may include a step of determining detailed indexes to be included in the second index based on the proportions of the first criterion, the second criterion, and the third criterion in the evaluation criteria.

[0015] Alternatively, the step of establishing an evaluation criterion for an artificial intelligence model based on the task intended by the user may include the steps of: acquiring information about the task intended by the user through user input; and inputting the information about the task intended by the user into a pre-trained criteria setting model to derive the evaluation criterion.

[0016] Alternatively, the step of establishing evaluation criteria for the artificial intelligence model based on the task intended by the user may include the steps of obtaining information about the task intended by the user through user input, and identifying evaluation categories and detailed criteria corresponding to the information about the task intended by the user in a pre-established database.

[0017] Alternatively, the step of constructing an artificial intelligence model that performs the task intended by the user based on the determined first and second indicators may include the steps of: learning an artificial intelligence model using the determined first indicator; evaluating the performance of the artificial intelligence model using the determined second indicator; and, if the performance of the evaluated artificial intelligence model satisfies the evaluation criteria, selecting the artificial intelligence model that satisfies the evaluation criteria as the artificial intelligence model that performs the task intended by the user.

[0018] To achieve the above object, one embodiment of the present disclosure provides a computer program stored on a computer-readable storage medium. The computer program, when executed by one or more processors, performs operations for constructing a medical AI model. Here, the operations may include: establishing an evaluation criterion for an AI model based on a task intended by a user; determining, based on the established evaluation criterion, a first index used in a loss calculation for training an AI model and a second index used in an evaluation calculation for selecting the trained model; and constructing an AI model that performs the task intended by the user based on the determined first and second indexes.

[0019] To achieve the above-described object, one embodiment of the present disclosure discloses a computing device for constructing a medical artificial intelligence model. The device may include a processor including at least one core and a memory including program code executable by the processor. Here, the processor may establish an evaluation criterion for an AI model based on a task intended by a user, determine a first index used in a loss calculation for training the AI ​​model and a second index used in an evaluation calculation for selecting the trained model based on the established evaluation criterion, and construct an AI model that performs the task intended by the user based on the determined first and second indexes. [Effects of the Invention]

[0020] The present disclosure can provide an environment in which models that can deliver optimal performance for the intended use and task can be stably developed by reflecting evaluation criteria in model learning and selection. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating a computational process for constructing a medical artificial intelligence model according to an embodiment of the present disclosure. [Figure 3] 1 is a flowchart illustrating a method for building a medical artificial intelligence model according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present disclosure. The embodiments presented in this disclosure are provided to enable those skilled in the art to use or practice the contents of the present disclosure. Therefore, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be embodied in various different forms and is not limited to the following embodiments.

[0023] Throughout the specification of the present disclosure, the same or similar reference numerals refer to the same or similar components. In addition, in order to clearly explain the present disclosure, reference numerals of parts that are not relevant to the explanation of the present disclosure may be omitted from the drawings.

[0024] The term "or" as used in this disclosure is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or otherwise clear from the context in this disclosure, "X uses A or B" should be understood to mean one of the natural inclusive permutations. For example, unless otherwise specified or otherwise clear from the context in this disclosure, "X uses A or B" can be interpreted as either X uses A, X uses B, or X uses both A and B.

[0025] The term "and / or" as used in this disclosure must be understood to indicate and include all possible combinations of one or more of the associated listed concepts.

[0026] The terms "comprises" and / or "comprising" as used in this disclosure should be understood to mean that the specified features and / or components are present. However, the terms "comprises" and / or "comprising" should not be understood to exclude the presence or addition of one or more other features, other components and / or combinations thereof.

[0027] In this disclosure, unless otherwise specified or clear from the context as referring to the singular form, the singular should generally be construed as including "one or more."

[0028] The term "nth (n is a natural number)" used in this disclosure can be understood as an expression used to distinguish components of the present disclosure from one another based on a predetermined criterion, such as functional, structural, or convenience of description. For example, in this disclosure, components that perform different functional roles can be classified as a first component or a second component. However, components that are substantially identical within the technical concept of the present disclosure but must be distinguished for convenience of description can also be classified as a first component or a second component.

[0029] The term "acquire" as used in this disclosure may be understood to mean not only receiving data from an external device or system via a wired or wireless communication network, but also generating data in an on-device form.

[0030] Meanwhile, the terms "module" or "unit" used in this disclosure may be understood to refer to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a portion thereof, hardware or a portion thereof, or a combination of software and hardware. Here, a "module" or "unit" may refer to a unit composed of a single element or a unit expressed as a combination or collection of multiple elements. For example, as a concept of connotation, a "module" or "unit" may refer to a hardware element or a collection of hardware elements of a computing device, an application program that performs a specific software function, a processing procedure implemented by executing software, or a collection of instructions for executing a program. Furthermore, as a broad concept, a "module" or "unit" may refer to a computing device itself that constitutes a system, or an application executed on a computing device. However, the above concepts are merely examples, and the concepts of "module" and "unit" may be defined in various ways within the scope of understanding of those skilled in the art based on the contents of this disclosure.

[0031] The term "model" as used in this disclosure may be understood as a system implemented using mathematical concepts and language to solve a specific problem, a collection of software units for solving a specific problem, or an abstract model of a processing process for solving a specific problem. For example, a neural network "model" may refer to a system implemented as a neural network that has problem-solving capabilities through learning. Here, a neural network may have problem-solving capabilities by optimizing parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network or a neural network ensemble in which multiple neural networks are combined.

[0032] The explanations of the above terms are intended to aid in understanding the present disclosure. Therefore, unless the above terms are explicitly stated as matters that limit the contents of the present disclosure, care should be taken not to use them in a way that limits the technical ideas of the contents of the present disclosure.

[0033] FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure.

[0034] The computing device 100 according to an embodiment of the present disclosure may be a hardware device or part of a hardware device that performs comprehensive data processing and calculations, or may be a software-based computing environment connected via a communication network. For example, the computing device 100 may be a server that performs intensive data processing functions and shares resources, or a client that shares resources by interacting with the server. The computing device 100 may also be a cloud system in which multiple servers and clients interact with each other to comprehensively process data. The above description is merely an example of a type of computing device 100, and various types of computing device 100 may be configured within the scope of what one skilled in the art would understand based on the contents of this disclosure.

[0035] 1, a computing device 100 according to an embodiment of the present disclosure may include a processor 110, a memory 120, and a network unit 130. However, since FIG. 1 is merely an example, the computing device 100 may include other components for implementing a computer environment. Also, the computing device 100 may include only some of the disclosed components.

[0036] The processor 110 according to an embodiment of the present disclosure may be understood as a component including hardware and / or software for performing computing operations. For example, the processor 110 may read a computer program to perform data processing for machine learning. The processor 110 may process operations such as input data processing for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. The processor 110 for performing such data processing may include a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. The types of processor 110 described above are merely examples, and various types of processor 110 may be configured within the scope of what one skilled in the art would understand based on the present disclosure.

[0037] The processor 110 may establish evaluation criteria based on information about a task intended by a user. The processor 110 may generate evaluation criteria based on information about the task acquired through user input. For example, the processor 110 may generate a user interface for receiving information about the task intended by a user. Here, the information input through the user interface may be basic information necessary for establishing evaluation criteria, such as the type of disease or the type of task, or the evaluation criteria themselves. The processor 110 may derive evaluation criteria by analyzing the basic information included in the user input based on a pre-established database or a pre-trained standard setting model. Here, the database may be a structured data set in which evaluation classifications, such as accuracy, and detailed criteria, such as AUROC (area under ROC), are matched for each task. The standard setting model may be a neural network model pre-trained to derive evaluation classifications and detailed criteria based on information about the task intended by a user. The standard setting model may be trained through supervised learning, or may be trained through unsupervised learning or self-supervised learning depending on the neural network structure or parameter configuration. The processor 110 may also generate the evaluation criteria by obtaining the evaluation criteria themselves from user input.

[0038] The processor 110 may determine the indices used for training and selecting an AI model based on the established evaluation criteria. The processor 110 may generate model learning indices and selection indices that can reflect the evaluation criteria through user input. The processor 110 may analyze the evaluation criteria and derive learning indices and evaluation indices corresponding to the evaluation criteria. For example, when an evaluation criterion is generated, the processor 110 may generate a user interface that lists indices that must be configured in the model's learning operation and selection operation, respectively, to reflect the evaluation criterion. When a user input for selecting an index is acquired through the user interface, the processor 110 may select the index according to the user input and determine the learning index and evaluation index. When an evaluation criterion is generated without user input, the processor 110 may analyze the evaluation criterion itself to generate the model's learning indices and selection indices. The processor 110 may determine the model's learning indices and selection indices according to the evaluation categories and detailed criteria included in the evaluation criterion based on a pre-established database or a pre-trained index determination model. Here, the database may be a data set in which learning indicators and selection indicators are structured along with evaluation classifications such as accuracy for each task and detailed criteria such as AUROC (area under ROC). The indicator determination model may be a neural network model pre-trained to derive learning indicators and selection indicators based on the evaluation classifications and detailed criteria. The indicator determination model may be trained by supervised learning, but may also be trained by unsupervised learning or self-supervised learning depending on the neural network structure or parameter configuration.

[0039] The processor 110 may construct an AI model using a learning index and a selection index derived based on the evaluation criteria. The processor 110 may train the AI ​​model using the learning index derived based on the evaluation criteria. The processor 110 may then evaluate the performance of the trained AI model using the selection index derived based on the evaluation criteria to determine whether the model performs the task intended by the user. If the performance evaluation does not satisfy the evaluation criteria based on the selection index, the processor 110 may readjust the learning conditions and parameters of the trained AI model to perform additional learning, or train a new model. If the performance evaluation satisfies the evaluation criteria based on the selection index, the processor 110 may select the model as the AI ​​model to perform the task intended by the user. Here, satisfying the evaluation criteria based on the selection index in the performance evaluation may be understood as the evaluation result being equal to or greater than a pre-set threshold. The threshold may be adjusted by the user or may be dynamically adjusted based on the task intended by the user.

[0040] As described above, the computation for constructing an AI model performed by the processor 110 of the present disclosure can reflect evaluation criteria tailored to user intentions in the model learning and selection process. Therefore, this computation minimizes the energy consumed in adjusting learning conditions and parameters based on the evaluation results, and can efficiently use resources required to develop a model that satisfies evaluation criteria that may change from time to time. In other words, the present disclosure can provide an environment in which models optimized for specific purposes and domains can be stably developed through such resource efficiency.

[0041] The memory 120 according to an embodiment of the present disclosure may be understood as a component including hardware and / or software for storing and managing data processed by the computing device 100. That is, the memory 120 may store any type of data generated or determined by the processor 110 and any type of data received by the network unit 130. For example, the memory 120 may include at least one type of storage medium selected from the group consisting of flash memory, hard disk, multimedia card micro, card-type memory, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. The memory 120 may also include a database system that manages data in a predetermined manner. The types of memory 120 described above are merely examples, and various configurations of the memory 120 are possible within the scope of what would be understood by one skilled in the art based on the present disclosure.

[0042] The memory 120 may structure and organize and manage data, data combinations, and program code executable by the processor 110 required for the processor 110 to perform calculations. For example, the memory 120 may store medical data received via the network unit 130 (described below). The memory 120 may store program code for causing an artificial intelligence model to receive medical data and perform learning, program code for causing a neural network model to receive electrocardiogram data and perform inference according to the intended use of the computing device 100, and processed data generated by executing the program code.

[0043] The network unit 130 according to an embodiment of the present disclosure may be understood as a component that transmits and receives data via any type of known wired or wireless communication system. For example, the network unit 130 may transmit and receive data using a wired or wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), 5th generation mobile communication (5G), ultra wide-band, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity (WiFi), near field communication (NFC), or Bluetooth. The above-described communication systems are merely examples, and various wired or wireless communication systems for transmitting and receiving data by the network unit 130 may be applied in addition to the above examples.

[0044] The network unit 130 may receive data necessary for the processor 110 to perform calculations via wired or wireless communication with any system or any client. The network unit 130 may also transmit data generated by calculations by the processor 110 via wired or wireless communication with any system or any client. For example, the network unit 130 may receive medical data by communicating with a database in a hospital environment, a cloud server that performs tasks such as medical data standardization, a client such as a smart watch, or a medical computing device. The network unit 130 may transmit output data of the neural network model, and intermediate data and processed data derived during the calculation process of the processor 110, via communication with the database, server, client, or medical computing device.

[0045] FIG. 2 is a block diagram illustrating a calculation process for constructing a medical artificial intelligence model according to one embodiment of the present disclosure.

[0046] Referring to FIG. 2 , a computing device 100 according to an embodiment of the present disclosure may establish an evaluation criterion 20 for an AI model based on task information 10 reflecting a purpose and a domain of use. The computing device 100 may acquire the task information 10 through user input. The task information 10 may be basic information indicating which task the AI ​​model will perform for which disease, or information about the evaluation criterion itself based on the basic information. The computing device 100 may analyze the acquired task information 10 to generate the evaluation criterion 20 for determining whether the AI ​​model can perform appropriate performance according to the purpose and domain of use. If the task information 10 includes the basic information, the computing device 100 may derive the evaluation criterion 20 through keyword matching analysis, AI-based analysis, etc. included in the basic information. If the task information 10 includes information about the evaluation criterion itself, the computing device 100 may determine the evaluation criterion 20 based on the task information 10.

[0047] The evaluation criteria 20 established by the computing device 100 may include at least one of a first criterion 21 for the accuracy of the AI ​​model, a second criterion 25 for the uncertainty of the AI ​​model's output, or a third criterion 29 for the correlation between the AI ​​model's output and a biometric value for determining whether or not a disease is present in a task intended by a user. The first criterion 21 may be a criterion for determining how accurately the AI ​​model performed the learned task. For example, indicators for evaluating the first criterion 21 may include AUROC, F1 Score, precision, sensitivity, recall, etc. The second criterion 25 may be a criterion for indicating the uncertainty of the results of a task performed by the AI ​​model. The second criterion 25 may be a criterion for evaluating whether, when the AI ​​determines that a specific disease has a 60% probability based on 100 people, 60 out of 100 people actually developed the specific disease. For example, indicators for evaluating the second criterion 25 may include adaptive calibration error (ACE) and expected calibration error (ECE). Each disease has a continuous value measurable on the body that can be used to determine whether or not a disease has occurred. The third criterion 29 may be a criterion for determining how closely a biometric value determining the presence or absence of a disease included in a task correlates with the outcome of a task performed by an AI model. For example, if the task intended by the user is to predict left ventricular systolic dysfunction, the third criterion 29 may be a criterion for calculating the correlation between the left ventricular ejection fraction (EF), a biometric value that can determine the presence or absence of left ventricular systolic dysfunction, and the result of the AI ​​model predicting left ventricular systolic dysfunction. If the task intended by the user is to predict myocardial infarction, the third criterion 29 may be a criterion for calculating the correlation between the troponin value, a biometric value that can determine the presence or absence of myocardial infarction, and the result of the AI ​​model predicting myocardial infarction.

[0048] Meanwhile, there may be multiple biometric measurements used to determine the presence or absence of a disease. For example, troponin as well as creatine kinase may be used as a biometric measurement to determine the presence or absence of myocardial infarction. Therefore, there may be at least one biometric measurement used as the third criterion 29.

[0049] The computing device 100 may configure the evaluation criteria 20 to include at least one of the first criterion 21, the second criterion 25, or the third criterion 29 according to the task information 10. For example, the computing device 100 may analyze the task information 10 to determine the proportions of the first criterion 21, the second criterion 25, and the third criterion 29 in the evaluation criteria 20. If the task information 10 includes a prediction of left ventricular systolic dysfunction, the computing device 100 may determine the proportions of the first criterion 21, the second criterion 25, and the third criterion 29 in the evaluation criteria 20 to be 4:3:3. Here, the computing device 100 may determine the proportions based on user input, or may determine the proportions through database rule-based analysis, artificial intelligence-based analysis, etc.

[0050] Referring to FIG. 2 , the computing device 100 may configure a first index 30 used in a loss calculation for training an AI model and a second index 40 used in an evaluation calculation for selecting a trained model based on the established evaluation criterion 20. Once the proportions of the first criterion 21, the second criterion 25, and the third criterion 29 are determined, the computing device 100 may determine a loss function included in the first index so as to calculate a correlation according to the third criterion 29. Because the third criterion 29 is a criterion specific to the intended use and domain, the computing device 100 may configure the loss function corresponding to the first index 30 to include a term for calculating the third criterion 29 so that the AI ​​model satisfies the third criterion 29 well. For example, if the task information 10 includes a prediction of left ventricular systolic dysfunction, the biological measurement value for determining the presence or absence of the disease in the third criterion 29 is determined by the left ventricular ejection fraction. Therefore, the computing device 100 may configure a left ventricular ejection fraction regression loss function as the loss function included in the first index. In the case where there are multiple biometric measurements to determine whether or not the third criterion 29 is present, the computing device 100 may generate a loss function included in the first index by constructing and combining loss functions corresponding to each of the multiple biometric measurements.

[0051] Furthermore, once the proportions of the first criterion 21, the second criterion 25, and the third criterion 29 are determined, the computing device 100 may determine detailed indexes included in the second index 40 and the proportion of each detailed index in the second index 40. The detailed indexes included in the second index 40 may be configured for each evaluation criterion. For example, if the task information 10 includes a prediction of left ventricular systolic dysfunction, the computing device 100 may configure the detailed index of the second index 40 for evaluating the first criterion 21 as a combination of F1 score, AUROC, and AUPRC (area under the precision recall curve). Furthermore, the computing device 100 may configure the detailed index of the second index 40 for evaluating the second criterion 25 as a combination of ACE and ECE. The form of the combination can be expressed by the following equation (1).

[0052]

number

[0053] The computing device 100 may select a left ventricular ejection fraction regression loss function as a detailed index for evaluating the third criterion 29. When there are multiple biometric measurements for determining the presence or absence of the disease of the third criterion 29, the computing device 100 may configure the detailed index for evaluating the third criterion 29 as a combination of the multiple biometric measurements. Here, the combination may be a simple summation of detailed indexes corresponding to each of the multiple biometric measurements, or a combination that generates a single integrated detailed index using a predetermined formula. The computing device 10 may configure the detailed index of the second index 40 as described above through analysis based on database rules, analysis based on artificial intelligence, etc. The computing device 100 may then configure the proportions of the detailed indexes of the second index 40 configured for each evaluation criterion to match the proportions of the first criterion 21, the second criterion 25, and the third criterion 28.

[0054] The computing device 100 may use the first index 30 in a calculation process in which the artificial intelligence model learns the task intended by the user. In other words, the computing device 100 may use the first index 30 in a loss calculation to train the artificial intelligence model to perform the task intended by the user. The computing device 100 may use the second index 40 in a calculation process in which the performance of the model after training is completed. In other words, the computing device 100 may use the second index 40 to evaluate whether the artificial intelligence model performed the task intended by the user in accordance with the evaluation criterion 20. If the performance of the artificial intelligence model evaluated using the second index 40 satisfies the evaluation criterion 20, the computing device 100 may select the model that satisfies the evaluation criterion 20 as the artificial intelligence model to perform the task intended by the user. If the performance of the artificial intelligence model evaluated using the second index 40 does not satisfy the evaluation criterion 20, the computing device 100 may perform additional training on the model that does not satisfy the evaluation criterion, perform training on a new model, or reconfigure the evaluation criterion 20 or the first index 30 and second index 40. Here, whether the evaluation criterion 20 is satisfied can be determined based on whether the evaluation value calculated by the second index 40 is equal to or greater than a preset threshold. For example, if the evaluation value calculated by the second index 40 is equal to or greater than a preset threshold, the computing device 100 can determine that the performance of the AI ​​model satisfies the evaluation criterion 20. If the evaluation value calculated by the second index 40 is less than the preset threshold, the computing device 100 can determine that the performance of the AI ​​model does not satisfy the evaluation criterion 20. Meanwhile, the threshold can be adjusted by user input or dynamically adjusted according to a rule determined depending on the type of detailed index included in the second index 40.

[0055] FIG. 3 is a flowchart illustrating a method for constructing a medical artificial intelligence model according to one embodiment of the present disclosure.

[0056] Referring to FIG. 3, a computing device 100 according to an embodiment of the present disclosure may establish evaluation criteria for an artificial intelligence model based on a task intended by a user (S100). The computing device 100 may acquire information about the task intended by the user through user input. The computing device 100 may then generate evaluation criteria by analyzing the information about the task. For example, if the computing device 100 is a client in a cloud system, the computing device 100 may receive user input through a user interface implemented by an input / output unit and generate information about the task intended by the user. If the computing device 100 is a server, the computing device 100 may receive user input through wired or wireless communication with the client and generate information about the task intended by the user. The computing device 100 may match, store, and manage information about the task intended by the user with evaluation criteria through a pre-defined database. Thus, the computing device 100 may identify evaluation categories and detailed criteria corresponding to information about the task intended by the user from the pre-defined database. The computing device 100 may determine the identified evaluation categories and detailed criteria as evaluation criteria by itself, or may determine the evaluation criteria through selection by user input. The computing device 100 may also derive evaluation criteria by inputting information about the task intended by the user into a pre-trained criteria setting model. Here, the criteria setting model may be a neural network model pre-trained based on data labeled with evaluation criteria matching the task intended by the user.

[0057] The computing device 100 may determine a first index used in a loss calculation for training an AI model and a second index used in an evaluation calculation for selecting a trained model according to the evaluation criteria established in step S100 (S200). The computing device 10 may determine a loss function included in the first index so as to calculate a correlation according to a third criterion for the correlation between the output of the AI ​​model and a biometric value for determining the presence or absence of a disease included in the task intended by the user, among the evaluation criteria. The computing device 100 may determine detailed indexes included in the second index according to the proportion of the first criterion for the accuracy of the AI ​​model, the second criterion for the uncertainty of the output of the AI ​​model, and the third criterion in the evaluation criteria.

[0058] The computing device 100 may construct an artificial intelligence model that performs the task intended by the user based on the first and second indicators determined in step S200 (S300). The computing device 100 may perform training of the artificial intelligence model using a loss function included in the first indicator. The computing device 100 may then evaluate the performance of the model trained using the loss function included in the first indicator, using the detailed indicators included in the second indicator. If the performance of the trained model matches the task originally intended by the user, the computing device 100 may select the model as the model that performs the task intended by the user.

[0059] The various embodiments of the present disclosure described above can be combined with additional embodiments and can be modified within the scope that can be understood by those skilled in the art from the above detailed description. The embodiments of the present disclosure are illustrative in all respects and should not be construed as limiting. For example, each component described as a single type can also be implemented in a distributed form, and similarly, components described as distributed can also be implemented in a combined form. Therefore, all modifications and variations derived from the meaning, scope, and equivalent concepts of the claims of the present disclosure should be construed as being within the scope of the present disclosure.

Claims

1. A method for constructing a medical artificial intelligence model, performed by a computing device including at least one processor, comprising: Establishing an evaluation criterion for the artificial intelligence model based on a task intended by a user; determining a first index used in a loss calculation for training an artificial intelligence model and a second index used in an evaluation calculation for selecting the trained model according to the established evaluation criterion; constructing an artificial intelligence model that performs the task intended by the user based on the determined first and second indicators; A method comprising:

2. The method of claim 1, wherein the evaluation criteria include at least one of a first criterion for the accuracy of the artificial intelligence model, a second criterion for the uncertainty of the output of the artificial intelligence model, or a third criterion for the correlation between the output of the artificial intelligence model and biometric measurements that determine the presence or absence of a disease included in the task intended by the user.

3. 3. The method of claim 2, wherein the user's intended task is prediction of left ventricular systolic dysfunction (LVSD), and the biometric value determining the presence or absence of the disease is left ventricular ejection fraction (EF).

4. The step of establishing an evaluation standard for the artificial intelligence model based on the task intended by the user includes: The method of claim 2 , further comprising determining the percentages of the evaluation criteria that the first criterion, the second criterion, and the third criterion account for based on a task intended by the user.

5. 5. The method of claim 4, wherein when the task intended by the user is prediction of left ventricular systolic dysfunction (LVSD), the ratio of the first criterion, the second criterion, and the third criterion in the evaluation criteria is determined to be 4:3:

3.

6. determining a first index used in a loss calculation for training an artificial intelligence model and a second index used in an evaluation calculation for selecting the trained model according to the established evaluation criterion, The method of claim 2 , further comprising determining a loss function included in the first index so that the correlation according to the third criterion can be calculated.

7. 7. The method of claim 6, wherein when the user's intended task is prediction of left ventricular systolic dysfunction (LVSD), the loss function included in the first index comprises a left ventricular ejection fraction (EF) regression loss function.

8. determining a first index used in a loss calculation for training an artificial intelligence model and a second index used in an evaluation calculation for selecting the trained model according to the established evaluation criterion, The method of claim 2 , further comprising determining detailed indicators included in the second indicator based on the proportions of the first criterion, the second criterion, and the third criterion in the evaluation criteria.

9. The step of establishing an evaluation criterion for the artificial intelligence model based on the task intended by the user includes: obtaining information about a task intended by the user through user input; deriving the evaluation criteria by inputting information about the task intended by the user into a pre-trained criteria setting model; The method of claim 1 , comprising:

10. The step of establishing an evaluation criterion for the artificial intelligence model based on the task intended by the user includes: obtaining information about a task intended by the user through user input; identifying, in a pre-established database, assessment categories and detailed criteria corresponding to information about the user's intended task; The method of claim 1 , comprising:

11. and constructing an artificial intelligence model that performs the task intended by the user based on the determined first and second indicators, training an artificial intelligence model using the determined first index; evaluating the performance of the artificial intelligence model using the determined second indicator; If the performance of the evaluated artificial intelligence model satisfies the evaluation criterion, selecting the artificial intelligence model that satisfies the evaluation criterion as an artificial intelligence model that will perform the task intended by the user; The method of claim 1 , comprising:

12. A computer program stored on a computer-readable storage medium, comprising: The computer program, when executed by one or more processors, performs an operation for constructing a medical artificial intelligence model, The operation is Establishing an evaluation criterion for the AI ​​model based on a task intended by a user; An operation of determining a first index used in a loss calculation for training an artificial intelligence model and a second index used in an evaluation calculation for selecting the trained model according to the established evaluation criterion; An operation of constructing an artificial intelligence model that performs the task intended by the user based on the determined first and second indicators; a computer program comprising:

13. A computing device for constructing a medical artificial intelligence model, comprising: a processor including at least one core; a memory containing program code executable by the processor; Including, The processor: Establish evaluation criteria for the AI ​​model based on the task intended by the user; determining a first index to be used in a loss calculation for training an artificial intelligence model and a second index to be used in an evaluation calculation for selecting the trained model according to the established evaluation criterion; An apparatus that constructs an artificial intelligence model that performs the task intended by the user based on the determined first and second indicators.

Images