Method, program, and apparatus for establishing artificial intelligence model for medical treatment

Abstract

The embodiment of the invention relates to a method, a program and a device which are executed by a computing device and used for establishing an artificial intelligence model for medical treatment. The method comprises the following steps of: making an evaluation reference of an artificial intelligence model on the basis of a task drawn by a user; determining a first index used in a loss calculation for learning of an artificial intelligence model and a second index used in an evaluation calculation for selecting the learned model according to the established evaluation criterion; and on the basis of the determined first index and second index, establishing an artificial intelligence model for executing the task drawn by the user.

Description

Technical Field

[0001] The present disclosure relates to deep learning technology in the medical field, and specifically, to a method for establishing an artificial intelligence model for medical treatment that meets the user's purpose. Background Art

[0002] In order to use an AI model, it is very important to let the AI ​​model learn well based on high-quality data that matches the domain of use. However, even if the AI ​​model is learned with high-quality data, it cannot be said with certainty that the AI ​​model will meet the user's purpose and performance of the domain of use. Therefore, it is very important to properly evaluate whether the learned model can exert good performance. In other words, the evaluation of the learned model can ensure the reliability, efficiency, and transparency of the model, and can also continuously improve the model according to the user's purpose and domain of use.

[0003] The benchmarks used to evaluate learning models are generally set independently of the learning conditions and parameters of the model. That is, in the process of building an artificial intelligence model, the evaluation is performed on the learned model, so the evaluation benchmark itself does not play a direct role in determining the learning conditions or parameters of the model. Moreover, the benchmarks used to evaluate learning models can include evaluation indicators that are specialized to the purpose or domain of use in addition to standard performance indicators, so it is very difficult to set the learning conditions and parameters of the model in a way that reflects such a flexible benchmark.

[0004] However, the more the AI ​​model performs in accordance with the evaluation criteria, the more likely it is that a model will be developed that can achieve the ultimate task that the AI ​​model is intended to achieve. Therefore, when it is necessary to include evaluation indicators that are specialized for the purpose or domain of use, as in the medical field, the evaluation indicators that are specialized for the purpose or domain of use need to be appropriately reflected throughout the model development process. Summary of the invention

[0005] Technical issues

[0006] The purpose of the present disclosure is to provide a method that derives an evaluation benchmark for a specified artificial intelligence model based on a task assigned by a user, configures parameters used for learning and selecting the model in a manner that conforms to the evaluation benchmark, and establishes an artificial intelligence model for medical use.

[0007] However, the technical problems to be solved by the present disclosure are not limited to the aforementioned problems, and other problems not mentioned above can be clearly understood from the following description.

[0008] Technical Solution

[0009] According to an embodiment of the present disclosure for achieving the aforementioned problem, a method for establishing an artificial intelligence model for medical treatment executed by a computing device is disclosed. The method includes the following steps: formulating an evaluation benchmark for the artificial intelligence model based on a task requested by a user; determining a first indicator used in a loss calculation for learning the artificial intelligence model and a second indicator used in an evaluation calculation for selecting the learned model based on the formulated evaluation benchmark; and establishing an artificial intelligence model that performs the task requested by the user based on the determined first and second indicators.

[0010] Alternatively, the evaluation benchmark may include at least one of the following benchmarks: a first benchmark regarding the accuracy of the artificial intelligence model; a second benchmark regarding the uncertainty of the output of the artificial intelligence model; or a third benchmark regarding the correlation between the output of the artificial intelligence model and a biological measurement value for determining whether the disease contained in the task assigned by the user has occurred.

[0011] Alternatively, if the task of the user is to predict left ventricular systolic dysfunction (LVSD), the biological measurement value for determining whether the disease occurs may be left ventricular ejection fraction (EF).

[0012] Alternatively, the step of formulating an evaluation benchmark for an artificial intelligence model based on the task intended by the user may include the following steps: determining the ratio of the first benchmark, the second benchmark, and the third benchmark in the evaluation benchmark based on the task intended by the user.

[0013] Alternatively, if the task attempted by the user is to predict left ventricular systolic dysfunction (LVSD), the ratio of the first criterion, the second criterion, and the third criterion in the evaluation criterion may be set to 4:3:3.

[0014] Alternatively, based on the established evaluation benchmark, the step of determining a first indicator used in a loss calculation for learning an artificial intelligence model and a second indicator used in an evaluation calculation for selecting the learned model may include the following steps: determining the loss function contained in the first indicator so that a correlation based on the third benchmark can be calculated.

[0015] Alternatively, if the task attempted by the user is to predict left ventricular systolic dysfunction (LVSD), the loss function included in the first indicator may include a left ventricular ejection fraction (EF) regression loss function.

[0016] Alternatively, based on the established evaluation benchmark, the steps of determining a first indicator used in a loss calculation for learning an artificial intelligence model and a second indicator used in an evaluation calculation for selecting the learned model may include the following steps: determining detailed indicators contained in the second indicator according to the ratio of the first benchmark, the second benchmark and the third benchmark in the evaluation benchmark.

[0017] Alternatively, the step of formulating an evaluation benchmark for an artificial intelligence model based on the task intended by the user may include the following steps: obtaining information about the task intended by the user based on the user's input; and inputting the information about the task intended by the user into a pre-learned benchmark setting model to derive the evaluation benchmark.

[0018] Alternatively, the step of formulating an evaluation benchmark for the artificial intelligence model based on the task intended by the user may include the following steps: obtaining information about the task intended by the user based on the user's input; and identifying evaluation categories and detailed benchmarks corresponding to the information about the task intended by the user in a pre-set database.

[0019] Alternatively, the step of establishing an artificial intelligence model that performs the task requested by the user based on the first and second indicators may include the following steps: using the first indicator to perform learning of the artificial intelligence model; using the second indicator to evaluate the performance of the artificial intelligence model; and if the performance of the evaluated artificial intelligence model meets the evaluation benchmark, selecting the artificial intelligence model that meets the evaluation benchmark as the artificial intelligence model that performs the task requested by the user.

[0020] One embodiment of the present disclosure that implements the aforementioned problem discloses a computer program stored in a computer-readable storage medium. The computer program performs multiple actions for establishing an artificial intelligence model for medical treatment when executed by one or more processors. At this time, the multiple actions may include the following actions: formulating an evaluation benchmark for the artificial intelligence model based on the task requested by the user; determining a first indicator used in the loss calculation for learning the artificial intelligence model and a second indicator used in the evaluation calculation for selecting the learned model based on the formulated evaluation benchmark; and establishing an artificial intelligence model that performs the task requested by the user based on the determined first and second indicators.

[0021] An embodiment of the present disclosure for achieving the aforementioned problem discloses a computing device for establishing an artificial intelligence model for medical treatment. The device includes: a processor including at least one core; a memory including a plurality of program codes executable by the processor; and a network unit for acquiring medical data. At this time, the processor formulates an evaluation benchmark for the artificial intelligence model based on the task requested by the user, determines a first indicator used in the loss calculation for learning the artificial intelligence model and a second indicator used in the evaluation calculation for selecting the learned model based on the formulated evaluation benchmark, and establishes an artificial intelligence model for performing the task requested by the user based on the determined first and second indicators.

[0022] Effects of the Invention

[0023] The present disclosure reflects the evaluation criteria in the learning and selection of the model, thereby providing an environment that can stably develop the following model, which can exert the best performance suitable for the purpose of use and the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a block diagram of a computing device according to an embodiment of the present disclosure.

[0025] Figure 2 It is a block diagram showing the calculation process of establishing an artificial intelligence model for medical treatment according to one embodiment of the present disclosure.

[0026] Figure 3 It is a flowchart showing a method for establishing an artificial intelligence model for medical treatment according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to enable a person of ordinary skill in the technical field to which the present disclosure belongs (hereinafter referred to as "a person of ordinary skill in the art") to easily implement the present disclosure, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. The embodiments disclosed in the present disclosure are used to enable a person of ordinary skill in the art to utilize or implement the contents of the present disclosure. Therefore, various variations of the embodiments of the present disclosure are obvious to a person of ordinary skill in the art. That is, the present disclosure can be implemented in various different forms and is not limited to the following embodiments.

[0028] The same or similar graphic symbols throughout the specification of the present disclosure represent the same or similar elements. In addition, the graphic symbols of the parts not related to the description of the present disclosure may be omitted in the drawings for the purpose of clearly explaining the present disclosure.

[0029] The term "or" used in the present disclosure does not mean an exclusive "or" but an inclusive "or". That is, if the present disclosure does not specify otherwise or its meaning is unclear in the context of the sentence, "X utilizes A or B" should be understood to mean one of the natural inclusive substitutions. For example, if the present disclosure does not specify otherwise or its meaning is unclear in the context of the sentence, "X utilizes A or B" can be interpreted as a situation where X utilizes A or a situation where X utilizes B or a situation where X utilizes both A and B.

[0030] The term “and / or” used in the present disclosure should be understood to refer to or include all possible combinations of more than one of the listed related concepts.

[0031] The terms "comprising" and / or "having" used in the present disclosure should be understood to mean the presence of specific features and / or elements. However, the terms "comprising" and / or "having" should be understood to not exclude the presence or addition of one or more other features, other elements and / or combinations thereof.

[0032] If the disclosure does not specifically specify or it is not clear from the context of the sentence that it indicates a singular form, the singular should generally be interpreted as including "one or more".

[0033] The term "Nth (N is a natural number)" used in the present disclosure can be understood as a way of expressing the elements of the present disclosure to be distinguished from each other according to a preset benchmark such as a functional viewpoint, a structural viewpoint, or for convenience of explanation. For example, elements that perform different functions in the present disclosure can be distinguished as first elements or second elements. However, elements that are essentially the same in the technical spirit of the present disclosure but need to be distinguished for the convenience of explanation can also be distinguished as first elements or second elements.

[0034] The term "acquire" used in the present disclosure may be understood to mean not only receiving data through a wired or wireless communication network with an external device or system, but also generating data in an on-device form.

[0035] On the other hand, the term "module" or "unit" used in the present disclosure can be understood as a term referring to an independent functional unit of processing computing resources such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, a combination of software and hardware, etc. In this case, a "module" or "unit" can be a unit composed of a single element, or a unit represented by a combination or set of multiple elements. For example, a "module" or "unit" as a protocol concept can refer to a hardware element or a set thereof of a computing device, an application program that executes a specific function of software, a process (procedure) implemented by executing software, or a set of instructions for executing a program, etc. Moreover, a "module" or "unit" as a broad concept can refer to the computing device itself that constitutes the system or an application program executed in the computing device, etc. However, the aforementioned concept is only an example, and the concept of "module" or "unit" can be variously defined within the scope that can be understood by ordinary technicians in the field based on the content of the present disclosure.

[0036] The term "model" used in the present disclosure can be understood as a system implemented using mathematical concepts and language to solve a specific problem, a collection of software units used to solve a specific problem, or an abstract model of a process used to solve a specific problem. For example, a neural network "model" can refer to the entire system implemented by a neural network that has the ability to solve problems through learning. In this case, the neural network can optimize the parameters of the connecting nodes (node) or neurons (neuron) through learning and thus have the ability to solve problems. A neural network "model" may include a single neural network or a neural network collection composed of multiple neural networks.

[0037] The explanation of the aforementioned terms is to help understand the present disclosure. Therefore, unless the aforementioned terms are clearly described as matters that limit the contents of the present disclosure, they are not used in the sense of limiting the technical spirit of the contents of the present disclosure.

[0038] Figure 1 is a block diagram of a computing device according to an embodiment of the present disclosure.

[0039] The computing device 100 of one embodiment of the present disclosure may be a hardware device or a part of a hardware device that performs comprehensive processing and calculation of data, or may be a computing environment based on software connected by a communication network. For example, the computing device 100 may be a server that is a main body that performs intensive data processing functions and shares resources, or may be a client that shares resources through interaction with a server. Moreover, the computing device 100 may also be a cloud system that performs comprehensive processing of data by means of interaction between multiple servers and multiple clients. The foregoing content is merely an example related to the type of computing device 100. The type of computing device 100 can be constructed in various ways within the scope that can be understood by ordinary technicians in the field based on the content of this disclosure.

[0040] See also Figure 1 , the computing device 100 of an embodiment of the present disclosure may include a processor 110, a memory 120 and a network unit 130. However, Figure 1 This is just an example, and the computing device 100 may include other elements for implementing a computing environment. Moreover, the computing device 100 may also include only a portion of the disclosed multiple elements.

[0041] The processor 110 of an embodiment of the present disclosure may be understood as a component unit including hardware and / or software for performing operations. For example, the processor 110 can read a computer program and then perform data processing for machine learning. The processor 110 can process operations such as processing input data for machine learning, extracting features for machine learning, and error calculation based on backpropagation. The processor 110 for performing data processing as described above may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an order-based semiconductor (ASICc: application specific integrated circuit) or a field programmable gate array (FPGA: field programmable gate array), etc. The types of the aforementioned processor 110 are merely an example, and the types of the processor 110 can be variously configured within the scope that can be understood by ordinary technicians in the field based on the content of the present disclosure.

[0042] The processor 110 can formulate an evaluation benchmark based on information about a task requested by a user. The processor 110 can generate an evaluation benchmark based on information about a task obtained based on input from a user. For example, the processor 110 can generate a user interface for inputting information about a task requested by a user. At this time, the information input through the user interface can be basic information required for formulating an evaluation benchmark, such as disease type, task type, etc., or it can be the evaluation benchmark itself. The processor 110 can derive an evaluation benchmark after analyzing the basic information contained in the user's input based on an established database or a pre-learned benchmark setting model. At this time, the database can be a structured data set that pairs evaluation classifications such as accuracy and detailed benchmarks such as the area under the receiver operating characteristic curve (AUROC) according to each task. Moreover, the benchmark setting model can be a neural network model that is pre-learned in a manner that derives evaluation classifications and detailed benchmarks after receiving input of information about the task requested by the user. The benchmark setting model can be learned based on supervised learning, but can also be learned based on unsupervised learning or self-supervised learning, etc., according to the neural network structure or parameter configuration. The processor 110 can also generate the evaluation criterion after obtaining the evaluation criterion itself from the user's input.

[0043] The processor 110 can determine the indicators for learning and selecting the artificial intelligence model based on the established evaluation benchmark. The processor 110 can generate learning indicators and selection indicators for the model that can reflect the evaluation benchmark based on the user's input. The processor 110 can analyze the evaluation benchmark and derive the learning indicators and evaluation indicators corresponding to the evaluation benchmark. For example, if an evaluation benchmark is generated, the processor 110 can generate the following user interface, which catalogs the multiple indicators that need to be configured in the learning calculation and selection calculation of the model in order to reflect the evaluation benchmark. If the user's input for selecting indicators is obtained through the user interface, the processor 110 can determine the learning indicators and evaluation indicators after selecting multiple indicators according to the user's input. Even if the evaluation benchmark is generated without user input, the processor 110 can analyze the evaluation benchmark itself and generate the learning indicators and selection indicators of the model. The processor 110 can determine the learning indicators and selection indicators of the model based on the evaluation classification and detailed benchmark contained in the evaluation benchmark based on the established database or the pre-learned indicator determination model. In this case, the database may be a data set that structures evaluation categories such as accuracy and detailed benchmarks such as the area under the receiver operating characteristic curve (AUROC) together with learning indicators and selection indicators according to each task. Furthermore, the indicator determination model may be a neural network model that is pre-learned in a manner that receives input of the evaluation categories and detailed benchmarks and derives the learning indicators and selection indicators. The indicator determination model can be learned based on supervised learning, but can also be learned based on unsupervised learning or self-supervised learning, etc., depending on the neural network structure or parameter configuration.

[0044] The processor 110 can establish an artificial intelligence model using learning indicators and selection indicators derived based on the evaluation benchmark. The processor 110 can use the learning indicators derived based on the evaluation benchmark to let the artificial intelligence model learn. In addition, the processor 110 uses the selection indicators derived based on the evaluation benchmark to evaluate the performance of the learned artificial intelligence model in order to determine whether it is a model that performs the task requested by the user. If the evaluation benchmark based on the selection indicator is not met in the performance evaluation, the processor 110 readjusts the learning conditions and parameters of the learned artificial intelligence model and further learns or allows a new model to learn. If the evaluation benchmark based on the selection indicator is met in the performance evaluation, the processor 110 can select the model as an artificial intelligence model that performs the task requested by the user. At this time, meeting the evaluation benchmark based on the selection indicator in the performance evaluation can be understood as the evaluation result being equivalent to or above the set critical value. The critical value can be adjusted by the user and can also be dynamically adjusted according to the task requested by the user.

[0045] According to the foregoing, the calculation performed by the processor 110 of the present disclosure for establishing an artificial intelligence model can reflect the evaluation criteria that cater to the user's intentions in the model learning and selection process. Therefore, the calculation can significantly reduce the energy wasted due to the operation of adjusting the learning conditions and parameters according to the evaluation results, and can efficiently use the resources required to develop a model that meets the evaluation criteria that may change each time. That is, the present disclosure can provide an environment that can stably develop models optimized for the purpose and domain of use through the high efficiency of the resources.

[0046] The memory 120 of an embodiment of the present disclosure may be understood as a hardware and / or software component unit that stores and manages data processed by the computing device 100. That is, the memory 120 can store any form of data generated or determined by the processor 110 and any form of data received by the network unit 130. For example, the memory 120 may include at least one storage medium in the form of flash memory type, hard disk type, multimedia card micro type, card memory, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, and optical disk. Moreover, the memory 120 may also include a database system that controls and manages data with a preset system. The aforementioned type of memory 120 is merely an example, and the type of memory 120 can be variously defined within a scope understandable to a person of ordinary skill in the art based on the contents of this disclosure.

[0047] The memory 120 can structure and organize the data required by the processor 110 for calculation, the combination of data, and the program code (code) executable by the processor 110, etc. For example, the memory 120 can store the medical data received through the network unit 130 described later. The memory 120 can store the program code that allows the artificial intelligence to learn after receiving the input of medical data, the program code that allows the neural network model to learn after receiving the input of electrocardiogram data, the program code that allows the neural network model to infer according to the purpose of use of the computing device 100 after receiving the input of medical data, and the processed data generated by the execution of the program code.

[0048] The network unit 130 of an embodiment of the present disclosure can be understood as a component unit that transmits and receives data through any form of a known wired wireless communication system. For example, the network unit 130 can use a wired wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband (WiBro), fifth generation mobile communication (5G), ultra wide-band wireless communication (ultra wide-band), ZigBee, radio frequency (RF), wireless LAN, wireless fidelity, near field communication (NFC) or Bluetooth to transmit and receive data. The aforementioned multiple communication systems are only examples, and various wired wireless communication systems for data transmission and reception of the network unit 130 can also be applied in addition to the aforementioned examples.

[0049] The network unit 130 can receive data required by the processor 110 for calculation through wired and wireless communication with any system or any client. In addition, the network unit 130 can send data generated by the calculation of the processor 110 through wired and wireless communication with any system or any client. For example, the network unit 130 can receive medical data through communication with a database in a hospital environment, a cloud server that performs tasks such as medical data standardization, a client such as a smart watch, or a medical computing device. The network unit 130 can send the output data of the neural network model and the intermediate data and processed data derived during the calculation of the processor 110 through communication with the aforementioned database, server, client, or computing device.

[0050] Figure 2 It is a block diagram showing the calculation process of establishing an artificial intelligence model for medical treatment according to one embodiment of the present disclosure.

[0051] See also Figure 2 , the computing device 100 of an embodiment of the present disclosure can formulate an evaluation benchmark 20 of an artificial intelligence model based on task information 10 that reflects the purpose of use and domain. The computing device 100 can obtain task information 10 based on user input. The task information 10 can be basic information indicating what kind of task the artificial intelligence model needs to perform for what disease, or it can be information about the evaluation benchmark itself based on the basic information. The computing device 100 can analyze the acquired task information 10 and generate the following evaluation benchmark 20, which is used to determine whether the artificial intelligence model can perform appropriate performance according to the purpose of use and domain. If the task information 10 contains basic information, the computing device 100 can derive the evaluation benchmark 20 through keyword matching analysis contained in the basic information, analysis based on artificial intelligence, etc. If the task information 10 contains information for the evaluation benchmark itself, the computing device 100 can determine the evaluation benchmark 20 based on the task information 10.

[0052] The evaluation benchmark 20 established by the computing device 100 may include at least one of the following benchmarks: a first benchmark 21, regarding the accuracy of the artificial intelligence model; a second benchmark 25, regarding the uncertainty of the output of the artificial intelligence model; or a third benchmark 29, which is a benchmark regarding the following correlation, which is the correlation between the output of the artificial intelligence model and the biological measurement value that determines whether the disease included in the task requested by the user occurs or not. The first benchmark 21 can be a benchmark for judging the accuracy of the artificial intelligence model in performing the learned task. For example, the indicators used to evaluate the first benchmark 21 may include the area under the receiver operating characteristic curve (AUROC), F1 score (F1 Score), precision, sensitivity, recall, etc. The second benchmark 25 can be a benchmark showing the degree of uncertainty of the result of the task performed by the artificial intelligence model. The second benchmark 25 can be a benchmark for evaluating whether 60 out of 100 people actually have a specific disease when the artificial intelligence judges that the possibility of the onset of a specific disease is 60% based on 100 people. For example, the index used to evaluate the second benchmark 25 may include an adaptive calibration error (ACE), an expected calibration error (ECE), etc. Each disease has a continuous value that can be measured from the body and can confirm whether it occurs or not. The third benchmark 29 can be a benchmark for judging how much correlation there is between the biological measurement value that determines whether the disease contained in the task occurs or not and the result of the task performed by the artificial intelligence model. For example, if the task that the user intends to perform is to predict left ventricular systolic dysfunction, the third benchmark 29 can be a benchmark for calculating the correlation between the left ventricular ejection fraction (EF) as a biological measurement value that can determine whether the left ventricular systolic dysfunction occurs or not and the result value of the left ventricular systolic dysfunction predicted by the artificial intelligence model. If the task that the user intends to perform is to predict myocardial infarction, the third benchmark 29 can be a benchmark for calculating the correlation between the troponin value as a biological measurement value that can determine whether the myocardial infarction occurs or not and the result value of the artificial intelligence model predicting myocardial infarction.

[0053] On the other hand, there may be a plurality of biological measurement values ​​for determining whether a disease has occurred. For example, as a biological measurement value for confirming whether a myocardial infarction has occurred, not only troponin but also creatinine kinase may be used. Therefore, there may be at least one biological measurement value used as the third reference 29.

[0054] The computing device 100 can construct the evaluation benchmark 20 in a manner including at least one of the first benchmark 21, the second benchmark 25, or the third benchmark 29 according to the task information 10. For example, the computing device 100 can determine the ratio of the first benchmark 21, the second benchmark 25, and the third benchmark 29 in the evaluation benchmark 20 after analyzing the task information 10. If the task information 10 includes a prediction of left ventricular systolic dysfunction, the computing device 100 can set the ratio of the first benchmark 21, the second benchmark 25, and the third benchmark 29 in the evaluation benchmark 20 to 4:3:3. At this time, the computing device 100 can determine the ratio based on the user's input or can determine the ratio through analysis based on database rules, analysis based on artificial intelligence, etc.

[0055] See also Figure 2 , the computing device 100 can configure the first index 30 used in the loss calculation for learning the artificial intelligence model and the second index 40 used in the evaluation calculation for selecting the learned model according to the established evaluation criterion 20. If the ratio of the first criterion 21, the second criterion 25 and the third criterion 29 is determined, the computing device 100 can determine the loss function included in the first index so that the correlation according to the third criterion 29 can be calculated. The third criterion 29 is a criterion specialized for the purpose of use and the domain, so the computing device 100 can form the loss function in such a way that the loss function equivalent to the first index 30 includes the term used to calculate the third criterion 29 so that the artificial intelligence model fully meets the third criterion 29. For example, if the task information 10 includes the prediction of left ventricular systolic dysfunction, the biological measurement value that determines whether the disease of the third criterion 29 occurs or not is determined to be the left ventricular ejection fraction, so the computing device 100 can configure the left ventricular ejection fraction regression loss function in the loss function included in the first index. If there are multiple biological measurement values ​​for determining whether the disease has occurred or not of the third criterion 29, the calculation device 100 can configure and combine the loss functions corresponding to the multiple biological measurement values ​​to generate the loss function included in the first index.

[0056] Furthermore, if the ratio of the first benchmark 21, the second benchmark 25, and the third benchmark 29 is determined, the computing device 100 can determine the detailed indicators included in the second index 40 and the ratio of each detailed indicator in the second index 40. The detailed indicators included in the second index 40 can be formed according to each evaluation benchmark. For example, if the task information 10 includes a prediction of left ventricular systolic dysfunction, the computing device 100 can form a detailed indicator of the second index 40 for evaluating the first benchmark 21 with a combination of the F1 score (F1 Score), the area under the receiver operating characteristic curve (AUROC), and the area under the PR curve (AUPRC, area under the precision recall curve). Furthermore, the computing device 100 can form a detailed indicator of the second index 40 for evaluating the second benchmark 25 with a combination of the adaptive calibration error (ACE) and the expected calibration error (ECE). The form of the combination can be expressed as in the following [Mathematical Formula 1].

[0057] [Mathematical formula 1]

[0058] Detailed metric for evaluating the first benchmark = (F1 score + AUROC + AUPRC) / 3

[0059] Detailed indicators for evaluating the second benchmark = ((1-ACE) + (1-ECE)) / 2

[0060] The computing device 100 can select the left ventricular ejection fraction regression loss function as a detailed index for evaluating the third benchmark 29. If there are multiple biological measurement values ​​for determining whether the disease occurs or not in the third benchmark 29, the computing device 100 can form a detailed index for evaluating the third benchmark 29 by combining the multiple biological measurement values. At this time, the combination can be a simple sum of multiple detailed indicators corresponding to the multiple biological measurement values, or a unified detailed indicator can be generated based on a preset mathematical formula. The computing device 10 can form the detailed index of the second index 40 as described above through analysis based on database rules, analysis based on artificial intelligence, etc. In addition, the computing device 100 can configure the ratio of the multiple detailed indicators of the second index 40 constructed according to each evaluation benchmark according to the ratio of the first benchmark 21, the second benchmark 25 and the third benchmark 29.

[0061] The computing device 100 can use the first indicator 30 in the calculation process of the artificial intelligence model learning the task requested by the user. That is, the computing device 100 can let the artificial intelligence model learn by using the first indicator 30 for loss calculation and letting the artificial intelligence model perform the task requested by the user. The computing device 100 can use the second indicator 40 in the calculation process of evaluating the performance of the learned model. That is, the computing device 100 can use the second indicator 40 to evaluate whether the artificial intelligence model has performed the task requested by the user according to the evaluation benchmark 20. If the performance of the artificial intelligence model evaluated using the second indicator 40 meets the evaluation benchmark 20, the computing device 100 can select the model that meets the evaluation benchmark 20 as the artificial intelligence model that performs the task requested by the user. If the performance of the artificial intelligence model evaluated using the second indicator 40 does not meet the evaluation benchmark 20, the computing device 100 can allow the model that does not meet the evaluation benchmark to further learn or allow a new model to learn or reconfigure the evaluation benchmark 200 or the first indicator 30 and the second indicator 40. At this time, whether the evaluation benchmark 20 is met can be determined based on whether the evaluation value calculated by the second indicator 40 is above the set critical value. For example, if the evaluation value calculated by the second indicator 40 is above the set critical value, the computing device 100 can determine that the performance of the artificial intelligence model meets the evaluation benchmark 20. If the evaluation value calculated by the second indicator 40 is lower than the set critical value, the computing device 100 can determine that the performance of the artificial intelligence model does not meet the evaluation benchmark 20. On the other hand, the critical value can be adjusted by the user's input, and can also be dynamically adjusted according to the set rules based on the type of detailed indicators included in the second indicator 40.

[0062] Figure 3 It is a flowchart showing a method for establishing an artificial intelligence model for medical treatment according to an embodiment of the present disclosure.

[0063] See also Figure 3, the computing device 100 of one embodiment of the present disclosure can formulate an evaluation benchmark of an artificial intelligence model based on the task requested by the user (step S100). The computing device 100 can obtain information about the task requested by the user based on the user's input. Moreover, the computing device 100 can generate an evaluation benchmark after analyzing the information about the task. For example, if the computing device 100 is a client on the cloud system, the computing device 100 can generate information about the task requested by the user after receiving the user's input through the user interface implemented by the input and output unit. If the computing device 100 is a server, the computing device 100 can generate information about the task requested by the user after receiving the user's input through wired and wireless communication with the client. The computing device 100 can store and manage the information about the task requested by the user and the evaluation benchmark after pairing them through a pre-set database. Therefore, the computing device 100 can identify the evaluation classification and detailed benchmark corresponding to the information about the task requested by the user in the pre-set database. The computing device 100 can set the identified evaluation classification and detailed benchmark itself as the evaluation benchmark, and can also determine the evaluation benchmark based on the selection based on the user's input. Furthermore, the computing device 100 can input information about the task requested by the user into a pre-learned benchmark setting model and derive the evaluation benchmark. In this case, the benchmark setting model can be a neural network model pre-learned based on the following data, which is data marked with the evaluation benchmark paired with the task requested by the user.

[0064] The computing device 100 can determine the first indicator used in the loss calculation for learning the artificial intelligence model and the second indicator used in the evaluation calculation for selecting the learned model based on the evaluation benchmark established by step S100 (step S200). The computing device 100 can determine the loss function included in the first indicator so that the correlation based on the third benchmark can be calculated, which is a benchmark related to the correlation between the output of the artificial intelligence model and the biological measurement value for determining whether the disease included in the task requested by the user has occurred in the evaluation benchmark. The computing device 100 can determine the detailed indicators included in the second indicator according to the first benchmark on the accuracy of the artificial intelligence model, the second benchmark on the uncertainty of the output of the artificial intelligence model, and the ratio of the third benchmark in the evaluation benchmark.

[0065] The computing device 100 can establish an artificial intelligence model that performs the task requested by the user based on the first indicator and the second indicator determined in step S200 (step S300). The computing device 100 can use the loss function included in the first indicator to perform learning of the artificial intelligence model. Moreover, the computing device 100 can use the detailed indicators included in the second indicator to evaluate the performance of the model learned using the loss function included in the first indicator. If the performance of the learned model meets the original user-requested task, the computing device 100 can select the model as the model that performs the user-requested task.

[0066] The various embodiments of the present disclosure described above can be combined with other different embodiments and can be changed within the scope understandable to a person of ordinary skill in the art according to the detailed description above. The embodiments of the present disclosure are illustrative in all respects and should be interpreted as not restrictive. For example, the elements described in a single overall form can be implemented in a dispersed manner, and similarly, the elements described in a dispersed form can also be implemented in a combined form. Therefore, all modifications and deformations derived from the meaning and scope of the claims of the present disclosure and their equivalent concepts should be interpreted as belonging to the scope of the present disclosure.

Claims

1. A method for establishing an artificial intelligence model for medical treatment, the method being performed by a computing device including at least one processor, comprising the following steps: Develop evaluation benchmarks for AI models based on the tasks that users are looking for; Determining, based on the established evaluation benchmark, a first indicator used in a loss calculation for learning an artificial intelligence model and a second indicator used in an evaluation calculation for selecting the learned model; and Based on the determined first and second indicators, an artificial intelligence model for executing the task requested by the user is established.

2. The method according to claim 1, characterized in that The evaluation criteria include at least one of the following criteria: The first benchmark is about the correctness of the AI ​​model; The second benchmark is about the uncertainty of the output of the AI ​​model; or The third benchmark is a benchmark regarding the following correlation, which is the correlation between the output of the artificial intelligence model and the biological measurement value for determining whether the disease included in the task assigned by the user has occurred.

3. The method according to claim 2, characterized in that If the task attempted by the user is to predict left ventricular systolic dysfunction, the biological measurement value for determining whether the disease has occurred is the left ventricular ejection fraction.

4. The method according to claim 2, characterized in that: The steps to develop an evaluation benchmark for an AI model based on the task requested by the user include the following steps: Based on the task requested by the user, the ratios of the first criterion, the second criterion, and the third criterion in the evaluation criterion are determined.

5. The method according to claim 4, characterized in that If the task attempted by the user is to predict left ventricular systolic dysfunction, the ratio of the first criterion, the second criterion, and the third criterion in the evaluation criterion is set to 4:3:

3.

6. The method according to claim 2, characterized in that The step of determining a first indicator used in a loss calculation for learning an artificial intelligence model and a second indicator used in an evaluation calculation for selecting the learned model according to the established evaluation benchmark comprises the following steps: The loss function included in the first indicator is determined so as to be able to calculate the correlation according to the third criterion.

7. The method according to claim 6, characterized in that If the task requested by the user is to predict left ventricular systolic dysfunction, the loss function included in the first indicator includes a left ventricular ejection fraction regression loss function.

8. The method according to claim 2, characterized in that: The step of determining a first indicator used in a loss calculation for learning an artificial intelligence model and a second indicator used in an evaluation calculation for selecting the learned model according to the established evaluation benchmark comprises the following steps: The detailed indicators included in the second indicator are determined according to the ratio of the first benchmark, the second benchmark and the third benchmark to the evaluation benchmark.

9. The method according to claim 1, characterized in that: The step of formulating an evaluation benchmark for an artificial intelligence model based on the task requested by the user comprises the following steps: Based on the user's input, obtaining information about the task that the user is attempting; as well as Information on the task attempted by the user is input into a previously learned benchmark setting model to derive the evaluation benchmark.

10. The method according to claim 1, characterized in that The step of formulating an evaluation benchmark for an artificial intelligence model based on the task requested by the user comprises the following steps: Based on the user's input, obtaining information about the task that the user is attempting; as well as An evaluation category and detailed criteria corresponding to the information about the task attempted by the user are identified in a predetermined database.

11. The method according to claim 1, characterized in that: The step of establishing an artificial intelligence model for executing the task requested by the user based on the determined first indicator and second indicator comprises the following steps: Performing learning of an artificial intelligence model using the determined first indicator; Evaluate the performance of the artificial intelligence model using the determined second indicator; as well as If the performance of the artificial intelligence model being evaluated meets the evaluation criteria, the artificial intelligence model that meets the evaluation criteria is selected as the artificial intelligence model that performs the task requested by the user.

12. A computer program, characterized in that The computer program is stored in a computer-readable storage medium, and when executed by one or more processors, the computer program performs a plurality of actions for establishing an artificial intelligence model for medical treatment. The multiple actions include the following actions: Develop evaluation benchmarks for AI models based on the tasks that users are looking for; Determining, based on the established evaluation benchmark, a first indicator used in a loss calculation for learning an artificial intelligence model and a second indicator used in an evaluation calculation for selecting the learned model; and Based on the determined first and second indicators, an artificial intelligence model for executing the task requested by the user is established.

13. A computing device for establishing an artificial intelligence model for medical treatment, characterized in that: include: a processor comprising at least one core; and a memory including a plurality of program codes executable by the processor, The processor formulates an evaluation benchmark for the artificial intelligence model based on the task requested by the user, determines a first indicator used in a loss calculation for learning the artificial intelligence model and a second indicator used in an evaluation calculation for selecting the learned model based on the formulated evaluation benchmark, and establishes an artificial intelligence model that performs the task requested by the user based on the determined first and second indicators.

Images