Method and apparatus for customizing machine learning model for medical image analysis

Abstract

A computing apparatus according to an aspect includes: a memory storing at least one program; and a processor configured to perform at least one operation by executing the at least one program, wherein the processor is further configured to acquire a pre-trained machine learning model generated at a first site, retrain the pre-trained machine learning model on the basis of information related to medical data collected from a second site, evaluate performance of the retrained machine learning model, and apply the retrained machine learning model on the basis of a result of the evaluation.

Description

DESCRIPTION Invention Title: METHOD AND APPARATUS FOR CUSTOMIZING MACHINE LEARNING MODEL FOR MEDICAL IMAGE ANALYSIS Technical Field The present disclosure relates to a method and apparatus for customizing a machine learning model. Particularly, the present disclosure relates to a method and apparatus for customizing a machine learning model for analysis of a medical image. Background Art Recently, technologies for predicting medical information regarding patients by analyzing medical images through machine learning models have been developed. In general, machine learning models may be trained in advance and delivered to users to suit purposes of the users (e.g., analysis of medical images, and the like). However, when pre-trained machine learning models are actually used, variations may occur in performance of the machine learning models, for respective users. For example, variations may occur in performance of machine learning models due to various causes, such as types of data held by users and bias of data used to train the machine learning models. Disclosure Technical Problem Provided are a method and apparatus for customizing a machine learning model so that the machine learning model may exhibit optimal performance for a purpose of a user. Provided is also a computer-readable recording medium having recorded thereon a program for causing a computer to execute the method described above. The technical problems to be solved are not limited to the technical problems as described above, and other technical problems may be present. Technical Solution A computing apparatus according to an aspect includes: a memory storing at least one program; and a processor configured to perform at least one operation by executing the at least one program, wherein the processor is further configured to acquire a pre-trained machine learning model generated at a first site, retrain the pre-trained machine learning model on the basis of information related to medical data collected from a second site, evaluate analysis performance of the retrained machine learning model for a medical image, and apply the retrained machine learning model on the basis of a result of the evaluation. A method of customizing a machine learning model according to another aspect includes: acquiring a pre-trained machine learning model generated at a first site; retraining the pre-trained machine learning model on the basis of information related to medical data collected from a second site; evaluating analysis performance of the retrained machine learning model for a medical image; and applying the retrained machine learning model on the basis of a result of the evaluation. A computer-readable recording medium according to another aspect includes a recording medium having recorded thereon a program for causing a computer to execute the method described above. Description of Drawings FIG. 1 is a view illustrating an example of customizing a machine learning model according to an embodiment. FIG. 2A is a block diagram illustrating an example of a user terminal according to an embodiment. FIG. 2B is a block diagram illustrating an example of a server according to an embodiment. FIG. 3 is a flowchart illustrating an example of a method of customizing a machine learning model, according to an embodiment. FIGS. 4A and 4B are views illustrating examples in which a processor acquires a pre-trained machine learning model, according to an embodiment. FIG. 5 is a view illustrating examples of data used for retraining for a pre- trained machine learning model, according to an embodiment. FIG. 6 is a view illustrating an example in which a processor retrains a pre-trained machine learning model, according to an embodiment. FIG. 7 is a view illustrating an example in which a processor retrains a pre-trained machine learning model, according to an embodiment. FIG. 8 is a view illustrating an example of an analysis process by a pre- trained machine learning model and a re-training process for the pre-trained machine learning model, according to an embodiment. FIG. 9 is a flowchart illustrating an example in which a processor evaluates performance of a retrained machine learning model and applies the same to a second server. FIG. 10 is a diagram illustrating an example of a system for analyzing a medical image. Best Mode A computing apparatus according to an aspect includes: a memory storing at least one program; and a processor configured to perform at least one operation by executing the at least one program, wherein the processor is further configured to acquire a pre-trained machine learning model generated at a first site, retrain the pre-trained machine learning model on the basis of information related to medical data collected from a second site, evaluate analysis performance of the retrained machine learning model for a medical image, and apply the retrained machine learning model on the basis of a result of the evaluation. Mode for Invention Although the terms used in embodiments have been selected from general terms currently widely used as much as possible, this may vary according to the intention of an engineer in the field or precedent, the emergence of new technologies, and the like. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meanings thereof will be described in detail in the relevant description. Therefore, the terms used in the description should be defined based on the meanings of the terms and the overall content of the description, rather than simply the names of the terms. When a part "includes" a component in the entire description, this means that it may further include other components rather than excluding the other components, unless otherwise stated. In addition, the terms "unit", "module", and the like described herein refer to units that process at least one function or operation and may be implemented as hardware or software, or a combination of hardware and software. In addition, the terms including ordinal numbers, such as "first" or "second" used in the description, may be used to describe various components, but the components should not be limited by the terms. The terms may be used to distinguish one component from another. Hereinafter, a "medical image" may refer to an image from which phenotypic features appearing in the human body may be extracted. For example, the medical image may include images of all modalities used in the medical field, such as a chest radiograph image, an X-ray image, a computed tomography (CT) image, a positron emission tomography (PET) image, a magnetic resonance imaging (MRI) image, an ultrasonic image, a sonography image, a functional magnetic resonance imaging (fMRI) image, a pathological slide image, a mammography (MMG) image, and a digital breast tomosynthesis (DBT) image. Hereinafter, the "pathological slide image" may refer to a whole slide image (WSI) including a high-resolution image of the whole pathological slide, and may also refer to a portion of the whole slide image (e.g., one or more patches or tiles). For example, the pathological slide image may be a digital image scanned via a scanning apparatus (e.g., a digital scanner or the like). The pathological slide image may be segmented into a tissue area in which a tissue collected from the human body is located and a background area, and information regarding a particular protein, cell, tissue and / or structure may be acquired from the tissue area. Hereinafter, "medical information" may include whether or not a lesion is present in a medical image, lesion information detected in the medical image, various medical findings other than the lesion, quality and metadata of the medical image (e.g., a modality, capturing information, and the like), quantitative information extracted from the medical image (e.g., the size, volume, ratio, number, and the like of the lesion), and the like. In addition, the medical information may refer to any medically meaningful information that may be extracted from the medical image. For example, the medical information may include at least one of information regarding an expression on the medical image, an immune phenotype, a genotype, a biomarker score, tumor purity, information regarding RNA, information regarding a tumor microenvironment, a treatment method of a cancer expressed in the medical image, survival information regarding a patient, and a treatment result. In addition, the medical information may include the area, location, and size of a particular tissue (e.g., a cancer tissue, a cancer stroma tissue, or the like) and / or a particular cell (e.g., a tumor cell, a lymphocyte cell, macrophage cells, endothelial cells, fibroblast cells, or the like) within the medical image, information regarding the diagnosis of cancer, information associated with the likelihood of a subject developing cancer, and / or medical conclusions associated with cancer treatment, but is not limited thereto. In addition, the medical information may include quantified figures obtained from the medical image, as well as information obtained by visualizing figures, predictive information according to the figures, image information, statistical information, and the like. For example, the medical information may be provided to a user terminal or output through a display apparatus. Hereinafter, a "biomarker" may refer to an objective and measurable biological index that may be used to predict a progression state or treatment outcome of a disease. For example, the biomarker may include expression levels and the like of HER2, TROP2, HER3, DLL3, MET, and FGFR2 proteins, are not limited to the examples described above, and may include all biological indexes such as various proteins expressed from all human genes. Hereinafter, embodiments are described in detail with reference to the accompanying drawings. However, the embodiments may be implemented in various different forms and are not limited to examples described herein. FIG. 1 is a view illustrating an example of customizing a machine learning model according to an embodiment. FIG. 1 illustrates a first server 11, a second server 21, and a user terminal 23. For example, the first server 11 may be arranged at a first site 10, and the second server 21 and the user terminal 23 may be arranged at a second site 20. For example, the first site 10 may be a company or research laboratory that analyzes or processes data. The first site 10 may research and develop a machine learning model 12 used to analyze data. The first site 10 may be a supplier that provides the machine learning model 12 to the second site 20. The first server 11 arranged at the first site 10 may train the machine learning model 12. For example, the first server 11 may train the machine learning model 12 by using pre-stored data. For example, data pre-stored in the first server 11 may include a medical image, medical information, and the like, but is not limited thereto. The machine learning model 12 trained at the first site 10 may be deployed in the second server 21 to be used by a user 24 of the second site 20. Here, the second server 21 may be an on-premise server arranged in a physical space of the second site 20. Alternatively, the second server 21 may be a cloud server that is not located in the physical space of the second site 20 but may be accessed by the user terminal 23 through a network. In the case where the second server 21 is the cloud server, the second server 21 or the user terminal 23 may be connected to the first server 11 through a wireless network. For example, the user 24 of the second site 20 may acquire desired information through a machine learning model 22 deployed in the second server 21. Assuming that the second site 20 is a hospital, the second server 21 in which the machine learning model 22 is deployed may analyze a medical image of a patient and output various types of medical information. Accordingly, the user 24 (e.g., a doctor, a pathologist, a radiologist, or the like) of the second site 20 may perform a diagnosis, treatment, observation of prognosis, and the like of the patient through the user terminal 23. Here, the user terminal 23 and the second server 21 may be connected to each other through a wired or wireless network. Here, the machine learning model 22 may be provided from the first site 10. In other words, the machine learning model 12 may be trained through the first server 11, and the trained machine learning model 22 may be deployed in the second server 21. Although the machine learning model 22 is pre-trained, the machine learning model 22 may not output an optimal result at the second site 20. For example, assuming that the second site 20 is a hospital or a research laboratory, data stored in the first server 11 may be different from data collected from the second site 20 in various factors (e.g., information of the patient, equipment provided in the hospital, and the like). Accordingly, the machine learning model 22 trained with the data stored in the first server 11 may fail to show the best performance in the second site 20. The machine learning model 22 according to an embodiment is retrained (or customized) on the basis of the data in the second server 21 after being pre- trained with the data in the first server 11. In other words, a pre-trained machine learning model is retrained through the data collected from the second site 20. Accordingly, the retrained machine learning model may output an optimal result at the second site 20. Accordingly, the user 24 may accurately perform diagnosis, observation of prognosis, medical research, and the like of the patient by using an output of the retrained machine learning model. Hereinafter, a computing apparatus may be the first server 11, the second server 21, or the user terminal 23. For example, the user terminal 23 may be an electronic apparatus provided with a display apparatus and a device (e.g., a keyboard, a mouse, or the like) that receives a user input and including a memory and a processor. In addition, the display apparatus may be implemented as a touch screen to perform a function of receiving a user input. For example, the user terminal 23 may correspond to a notebook PC, a desktop PC, a laptop, a tablet computer, a smartphone, or the like, but is not limited thereto. For example, the first server 11 or the second server 21 may be an apparatus that communicates with an external device (e.g., the user terminal 23 or the like). For example, the first server 11 or the second server 21 may be an apparatus that stores various types of data, including a medical image, a bitmap image corresponding to the medical image, information generated by analysis of the medical image (e.g., information regarding a lesion detected in the medical image, information regarding findings other than a lesion detected in the medical image, a medical image interpretation report, information regarding at least one object expressed in the medical image, information regarding at least one biomarker expression, medical information related to the medical image, or the like), quality information regarding the medical image, information regarding metadata, and information regarding a machine learning model used for analysis of the medical image. Alternatively, the first server 11 or the second server 21 may be an electronic apparatus including a memory and a processor and having a computational ability. The computing apparatus may identify biological factors (e.g., a lesion, medical findings, a cancer cell, an immune cell, a cancer area, and the like) expressed in a medical image by analyzing the medical image through a machine learning model. The biological factors may be used for a histological diagnosis of a disease, prediction of a disease prognosis, determination of treatment direction for the disease, and the like. Hereinafter, an example in which a machine learning model is retrained and the retrained machine learning model is applied to the second server 21 or the user terminal 23 is described with reference to FIGS. 2A to 10. As described above, the computing apparatus may be the first server 11, the second server 21, or the user terminal 23. Therefore, operations performed by the computing apparatus below may be performed by the first server 11, the second server 21, or the user terminal 23. Alternatively, some of the operations performed by the computing apparatus below may be performed by the first server 11 or the second server 21, and the rest may be performed by the user terminal 23. Hereinafter, examples of a user terminal and a server are described with reference to FIGS. 2A and 2B. FIG. 2A is a block diagram illustrating an example of a user terminal according to an embodiment. Referring to FIG. 2A, a user terminal 100 includes a processor 110, a memory 120, an input / output interface 130, and a communication module 140. For convenience of description, FIG. 2A illustrates only components related to the present disclosure. Accordingly, the user terminal 100 may further include other general-purpose components in addition to the components illustrated in FIG. 2A. In addition, it is obvious to those skilled in the art related to the present disclosure that the processor 110, the memory 120, the input / output interface 130, and the communication module 140 illustrated in FIG. 2A may be implemented as independent devices. Also, the user terminal 100 may be the user terminal 23 illustrated in FIG. 1. The processor 110 may process commands of a computer program by performing basic arithmetic, logic, and input / output operations. Here, the commands may be provided from the memory 120 or an external apparatus (e.g., the server 200 or the like). In addition, the processor 110 may generally control operations of other components included in the user terminal 100. In addition, at least one of operations of the processor 210, which are described below with reference to FIG. 2B, may be performed by the processor 110. The processor 110 may be implemented as an array of a plurality of logical gates, or may be implemented as a combination of a general-purpose microprocessor and a memory that stores a program executable by the microprocessor. For example, the processor 110 may include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, or the like. In some environments, the processor 110 may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like. For example, the processor 110 may refer to a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors combined with a digital signal processor (DSP) core, or any other combination of such components. The memory 120 may include any non-transitory computer-readable recording medium. As an example, the memory 120 may include a permanent mass storage device such as random access memory (RAM), read only memory (ROM), a disk drive, a solid state drive (SSD), or flash memory. As another example, the permanent mass storage device such as ROM, an SSD, flash memory, or a disk drive may be a separate permanent storage device that is distinguished from a memory. In addition, the memory 120 may store an operating system (OS) and at least one program code (e.g., a code for the processor 110 to perform an operation described below with reference to FIGS. 3 to 10). Software components described above may be loaded from a computer- readable recording medium that is separate from the memory 120. The separate computer-readable recording medium may be a recording medium that may be directly connected to the user terminal 100 and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. Alternatively, the software components may also be loaded into the memory 120 through the communication module 140 rather than the computer-readable recording medium. For example, the at least one program may be loaded into the memory 120 on the basis of a computer program that is installed by files provided through the communication module 140 by developers or a file distribution system that distributes installation files of applications (e.g., a computer program or the like for the processor 110 to perform operations described below with reference to FIGS. 3 to 10). The input / output interface 130 may be a unit for interfacing with a device (e.g., a keyboard, a mouse, or the like) for an input or output, which may be connected to the user terminal 100 or may be included in the user terminal 100. Although FIG. 2A illustrates the input / output interface 130 as a component configured separately from the processor 110, but the input / output interface 130 is not limited thereto and may also be configured to be included in the processor 110. The communication module 140 may provide a component or function for the server 200 and the user terminal 100 to communicate with each other through a network. In addition, the communication module 140 may provide a component or function for the user terminal 100 to communicate with another external device. For example, a control signal, a command, data, and the like, which are provided under control of the processor 110, may be transmitted to the server 200 and / or an external device through the communication module 140 and the network. Meanwhile, although not illustrated in FIG. 2A, the user terminal 100 may further include a display apparatus. Alternatively, the user terminal 100 may be connected to an independent display apparatus through a wired or wireless communication method to transmit and receive data to and from each other. For example, a medical image, analysis information regarding the medical image, medical information, additional information based on the medical information, and the like may be provided to the user 24. FIG. 2B is a block diagram illustrating an example of a server according to an embodiment. Referring to FIG. 2B, a server 200 includes a processor 210, a memory 220, and a communication module 230. For convenience of description, FIG. 2B illustrates only components related to the present disclosure. Accordingly, the server 200 may further include other general-purpose components in addition to the components illustrated in FIG. 2B. In addition, it is obvious to those skilled in the art related to the present disclosure that the processor 210, the memory 220, and the communication module 230 illustrated in FIG. 2B may be implemented as independent devices. Also, the server 200 may be the first server 11 or the second server 21 illustrated in FIG. 1. The processor 210 may process commands of a computer program by performing basic arithmetic, logic, and input / output operations. Here, the commands may be provided from the memory 120 or an external apparatus (e.g., the user terminal 100 or the like). In addition, the processor 210 may generally control operations of the other components included in the server 200. The processor 210 acquires a pre-trained machine learning model. The pre-trained machine learning model may be a machine learning model generated by the first server 11. For example, in the case where the server 200 is the first server 11 of FIG. 1, the processor 210 may read data of the pre-trained machine learning model from the memory 220 or receive data of the pre-trained machine learning model from the second server 21. In the case where the server 200 is the second server 21 of FIG. 1, the processor 210 may read data of the pre- trained machine learning model from the memory 220 or receive data of the pre- trained machine learning model from the first server 11. A particular example in which the processor 210 acquires the pre-trained machine learning model is described below with reference to operation 310 of FIG. 3. For example, the processor 210 may retrain the pre-trained machine learning model on the basis of information related to the second site 20. Here, the information related to the second site 20 may include information related to medical data collected from the second site 20, information generated by the second server 21 or stored in the second server 21, and the like. For example, the pre-trained machine learning model may be deployed in the second server 21. Accordingly, the information related to the second site 20 may also include data generated by the pre-trained machine learning model. For example, the processor 210 may retrain the pre-trained machine learning model on the basis of the information related to the medical data collected from the second site 20. The process described above may refer to a process of customizing the machine learning model to the second site 20. The processor 210 may acquire the medical data collected from the second site 20 from an external apparatus or the memory 220. The medical data collected from the second site 20 may include at least one of medical data related to a disease, diagnosis, or treatment of a patient stored in a database of the second site 20 and medical data related to a disease, diagnosis, or treatment acquired from a patient visiting the second site 20. The medical data collected from the second site 20 may be information related to a disease, diagnosis, or treatment of a patient, and may include at least one of a medical image, patient information, and medical information that may be extracted from the medical image. The processor 210 may customize the machine learning model to the second site 20 by using, for retraining for the pre-trained machine learning model, the medical image collected from the second site 20 and an annotation by a user on the corresponding medical image. The annotation by the user of the second site 20 may be considered as a ground truth optimized for the second site 20. The annotation by the user may include whether or not a lesion is present in the medical image, lesion information detected in the medical image, various medical findings other than the lesion, quality and metadata of the medical image (e.g., a modality, capturing information, and the like), quantitative information extracted from the medical image (e.g., the size, volume, ratio, number of lesions, and the like), and the like. Meanwhile, the present disclosure is not limited to an embodiment of updating a machine learning model through retraining for a pre-trained machine learning model. According to an embodiment, the processor 210 may newly construct a machine learning model by using at least one of the medical image collected from the second site 20, the annotation by the user on the medical image, and the medical information. According to an embodiment, the processor 210 may evaluate performance of at least one of the pre-trained machine learning model and the retrained machine learning model that analyzes the medical data collected from the second site 20, and determine whether or not to newly construct the machine learning model on the basis of the result of the evaluation. In the case where the performance of at least one of the pre-trained machine learning model and the retrained machine learning model is threshold performance or less, the processor 210 may determine to newly construct the machine learning model. Newly constructing the machine learning model may include a process of training the machine learning model by using only the medical data collected from the second site 20. The processor 210 may also use medical information other than an image form for retraining for the machine learning model. The processor 210 may use the result of diagnosis by a doctor in the second site 20 for retraining for the machine learning model. For example, the processor 210 may acquire medical information by analyzing a diagnosis report from the doctor on the medical image of the patient by using a large language model, and use the acquired medical information for retraining for the machine learning model. The processor 210 may retrain the pre-trained machine learning model on the basis of the medical image and medical information of the patient. The processor 210 may retrain the pre-trained machine learning model on the basis of data satisfying a certain criterion from among the medical data collected from the second site 20. According to various embodiments described above, the processor 210 may customize the pre-trained machine learning model to the second site 20 by using various model tuning methods. For example, the processor 210 may perform retraining optimized for the second site 20 by automatically tuning at least one hyperparameter of the pre-trained machine learning model or tuning a model architecture, without intervention of the user by using an automated machine learning (AutoML) technique. In addition, the processor 210 may retrain the pre-trained machine learning model on the basis of data generated by the pre-trained machine learning model. For example, the processor 210 may retrain the pre-trained machine learning model by using all of the data generated by the pre-trained machine learning model. Alternatively, the processor 210 may select data to be used for retraining from among the data generated by the pre-trained machine learning model. In addition, the processor 210 may retrain the pre-trained machine learning model by using the selected data. As an example, the processor 210 may analyze the medical image collected from the second site 20 via the pre-trained machine learning model. The user may customize the machine learning model to the second site 20 by using, for retraining for the pre-trained machine learning model, the result of the analysis by the pre-trained machine learning model. The processor 210 may retrain the pre-trained machine learning model on the basis of labeled data in which the result of analyzing the medical image by the pre-trained machine learning model is reflected. As another example, the processor 210 may analyze the medical image collected from the second site 20 via the pre-trained machine learning model. The user may perform an annotation task of accepting, modifying, or rejecting the result of the analysis by the pre-trained machine learning model. The annotation by the user of the second site 20 may be considered as a ground truth optimized for the second site 20. The processor 210 may customize the machine learning model to the second site 20 by using the annotation by the user for retraining for the pre-trained machine learning model. The processor 210 may retrain the pre- trained machine learning model on the basis of first labeled data in which the annotation by the user on the medical image analyzed by the pre-trained machine learning model is reflected. The processor 210 may retrain the pre-trained machine learning model on the basis of the first labeled data. Here, the first label data includes data in which the annotation by the user on the medical image analyzed by the pre- trained machine learning model is reflected. As another example, the processor 210 may retrain the pre-trained machine learning model by further including pseudo second labeled data in the first labeled data (i.e., labeled data or annotated data) in which feedback provided by the user is reflected. Here, the pseudo labeled data refers to data sampled by applying a data-based sampling technique (e.g., a class-aware balancing technique). For example, the first labeled data may indicate a negatively biased feedback (NBF) phenomenon. Therefore, the processor 210 may generate second labeled data by sampling unlabeled data that has the same pseudo labels as the first labeled data. As another example, the processor 210 may retrain the pre-trained machine learning model on the basis of medical information corresponding to the medical image analyzed by the pre-trained machine learning model. Here, the medical information may be acquired by analysis of data stored in a database of a medical information system (e.g., an electronic medical record (EMR), a hospital information service (HIS), an order communication system (OCS), a picture archiving communication system (PACS), or the like) by the machine learning model. The processor 210 may acquire the medical information by analyzing the data stored in the database of the medical information system by using the machine learning model. The processor 210 may retrain the pre-trained machine learning model on the basis of the medical information. Here, the machine learning model used to acquire the medical information from the data stored in the database of the medical information system may be different from a pre-trained machine learning model and a retrained machine learning model described in the present disclosure. As another example, the processor 210 may retrain the pre-trained machine learning model on the basis of data satisfying a certain criterion from among data generated by the pre-trained machine learning model. Here, the data generated by the pre-trained machine learning model may be received from an external apparatus. In the case where the server 200 is the second server 21 of FIG. 1, the external apparatus may be the first server 11 or the user terminal 23. In the case where the server 200 is the first server 11 of FIG. 1, the external apparatus may be the second server 21 or the user terminal 23. According to various embodiments described above, the processor 210 may customize the pre-trained machine learning model to the second site 20 by using various model tuning methods. For example, the processor 210 may perform retraining optimized for the second site 20 by automatically tuning at least one hyperparameter of the pre-trained machine learning model or tuning a model architecture, without intervention of the user by using an automated machine learning (AutoML) technique. The processor 210 may retrain the pre-trained machine learning model separately from a process in which the medical image is analyzed (e.g., inferred) by the pre-trained machine learning model. In other words, the process of analysis by the pre-trained machine learning model and a process of retraining for the pre-trained machine learning model may be performed independently of and in parallel with each other. Accordingly, the processor 210 may analyze the medical image by using each of the pre-trained machine learning model and the retrained machine learning model. The processor 210 may evaluate each of medical image analysis performance of the pre-trained machine learning model and medical image analysis performance of the retrained machine learning model. A particular example in which the processor 210 retrains the pre-trained machine learning model is described below with reference to operation 320 of FIG. 3. The processor 210 may evaluate performance of the retrained machine learning model. For example, the processor 210 may evaluate the performance of the retrained machine learning model on the basis of at least one of a first index for an area under the curve (AUC) of the retrained machine learning model, a second index for a distribution of results of analysis known in advance for the second site 20 compared to the results of analysis of medical images by the retrained machine learning model, and a third index for the results of analysis of the medical images by the pre-trained machine learning model compared to the results of analysis of the medical images by the retrained machine learning model. A particular example in which the processor 210 evaluates the performance of the retrained machine learning model is described below with reference to operation 330 of FIG. 3. The processor 210 may apply the retrained machine learning model on the basis of the result of the evaluation. For example, the processor 210 may apply the retrained machine learning model by any one of a first method by which the retrained machine learning model automatically replaces the pre-trained machine learning model and a second method by which the retrained machine learning model replaces the pre-trained machine learning model on the basis of a user input. A particular example in which the processor 210 applies the retrained machine learning is described below with reference to operation 340 of FIG. 3. An implementation example of the processor 210 is the same as the implementation example of the processor 110 described above with reference to FIG. 2A, and thus, a detailed description thereof is omitted. The memory 220 may store various types of data such as a medical image, and data generated according to an operation of the processor 210. In addition, the memory 220 may store an operating system (OS) and at least one program (e.g., a program needed for the processor 210 to operate, and the like). An implementation example of the memory 220 is the same as the implementation example of the memory 120 described above with reference to FIG. 2A, and thus, a detailed description thereof is omitted. The communication module 230 may provide a component or function for the server 200 and the user terminal 100 to communicate with each other through a network. In addition, the communication module 230 may provide a component or function for the server 200 to communicate with another external device. For example, a control signal, a command, data, and the like, which are provided under control of the processor 210, may be transmitted to the user terminal 100 and / or an external device (e.g., another server) through the communication module 140 and the network. FIG. 3 is a flowchart illustrating an example of a method of customizing a machine learning model, according to an embodiment. The method illustrated in FIG. 3 includes operations processed in a time series by the processors 110 and 210 illustrated in FIGS. 2A and 2B. Accordingly, although the above descriptions of the processors 110 and 210 are omitted below, the descriptions may also be applied to the method illustrated in FIG. 3. In addition, at least one of operations performed by the processor 210 below may be processed by the processor 110. In operation 310, the processor 210 acquires a pre-trained machine learning model. The pre-trained machine learning model refers to a model that is completely trained before being deployed in the second server 21. For example, the pre-trained machine learning model may be a model trained by the first server 11. The first server 11 may train a machine learning model by using data stored in a memory. The method of training is not limited to any one, and the machine learning model may be trained by various methods included in supervised learning or unsupervised learning. For example, the machine learning model may be used to generate medical information by analyzing a medical image. Here, the medical image is not limited to any one type, and may include images of all modalities used in the medical field. The pre-trained machine learning model is acquired to be retrained. For example, the machine learning model may be trained by the first server 11, and the trained machine learning model may be deployed and operated in the second server 21. Meanwhile, retraining of the pre-trained machine learning model may be performed by the first server 11 or may be performed by the second server 21. In other words, the pre-trained machine learning model may be retrained by a first computing apparatus of the first site 10 that generates the pre-trained machine learning model or a second computing apparatus of the second site 20 in which the pre-trained machine learning model is deployed. In other words, the processor 210 may be a processor of the first server 11 or a processor of the second server 21. Hereinafter, examples of a subject that acquires a pre-trained machine learning model for retraining are described with reference to FIGS. 4A and 4B. FIGS. 4A and 4B are views illustrating examples in which a processor acquires a pre-trained machine learning model, according to an embodiment. FIG. 4A illustrates an example in which a second server 21 acquires and retrains a pre-trained machine learning model 410. In other words, the second server 21 may acquire the pre-trained machine learning model 410 and generate a retrained machine learning model 420. Also, the processor 210 may be included in the second server 21. Hereinafter, the retrained machine learning model 420 being generated indicates that the pre-trained machine learning model 410 is updated by retraining. In other words, the retrained machine learning model 420 being generated indicates that the pre-trained machine learning model 410 is customized. The second server 21 may be a computing apparatus in which the pre- trained machine learning model 410 is deployed. Accordingly, the second server 21 does not need to receive the pre-trained machine learning model 410 from an external device. In other words, the processor 210 may read the pre-trained machine learning model 410 stored in the memory 220. In addition, the processor 210 may retrain the pre-trained machine learning model 410 to generate the retrained machine learning model 420. FIG. 4B illustrates an example in which a first server 11 acquires a pre- trained machine learning model 410 and retrains the pre-trained machine learning model 410. In other words, the first server 11 may acquire the pre-trained machine learning model 410 and generate a retrained machine learning model 420. Also, the processor 210 may be included in the first server 11. The first server 11 may be a computing apparatus that trains a machine learning model. In addition, the pre-trained machine learning model 410 may be deployed in a second server 21. As indicated by an arrow at the bottom of FIG. 4B, the processor 210 of the first server 11 may receive the pre-trained machine learning model 410 from the second server 21. In addition, the processor 210 may retrain the pre-trained machine learning model 410 to generate the retrained machine learning model 420. Also, as indicated by an arrow at the top of FIG. 4B, the processor 210 may transmit the retrained machine learning model 420 to the second server 21. Referring back to FIG. 3, in operation 320, the processor 210 retrains the pre-trained machine learning model. The processor 210 may retrain the pre-trained machine learning model on the basis of information related to the second site 20. For example, the information related to the second site 20 may include information related to medical data collected from the second site 20, information generated by the second server 21 or stored in the second server 21, or data generated by the pre- trained machine learning model. As an example, the processor 210 may retrain the pre-trained machine learning model on the basis of the information related to the medical data collected from the second site 20. For example, the processor 210 may retrain the pre-trained machine learning model on the basis of the medical data collected from the second site 20. Alternatively, the processor 210 may retrain the pre- trained machine learning model on the basis of the result of analyzing a medical image collected from the second site 20 by using the pre-trained machine learning model. As another example, the processor 210 may acquire medical information by analyzing the result of diagnosing the medical image collected from the second site 20 by a doctor by using a large language model. The processor 210 may retrain the pre-trained machine learning model on the basis of the medical image and the medical information. As another example, the processor 210 may retrain the pre-trained machine learning model on the basis of data satisfying a certain criterion from among the medical data collected from the second site 20. The processor 210 may customize the pre-trained machine learning model to the second site 20 by using various model tuning methods. For example, the processor 210 may automatically tune at least one hyperparameter of the pre-trained machine learning model or change a model architecture, without intervention of a user by using an automated machine learning (AutoML) technique. As another example, the processor 210 may select data to be used for retraining from among the data generated by the pre-trained machine learning model. In addition, the processor 210 may retrain the pre-trained machine learning model by using the selected data. As another example, the processor 210 may also retrain the pre-trained machine learning model by using all data generated by the pre-trained machine learning model. For example, the data generated by the pre-trained machine learning model may include various types of medical information output as the medical image is analyzed. In other words, the data used for retraining may include the medical image and the result (i.e., the medical information) of analyzing the medical image by the pre-trained machine learning model. Hereinafter, examples of data used for retraining for a pre-trained machine learning model are described with reference to FIG. 5. FIG. 5 is a view illustrating examples of data used for retraining for a pre- trained machine learning model, according to an embodiment. Referring to FIG. 5, the processor 210 may retrain a pre-trained machine learning model 410 by using data stored in a memory 510. Accordingly, a retrained machine learning model 420 may be generated. Here, the data stored in the memory 510 may include various types of medical information output as a medical image is analyzed by the pre-trained machine learning model 410. The processor 210 may use at least some of the data stored in the memory 510 for retraining for the pre-trained machine learning model 410. For example, the memory 510 may be the memory 220 illustrated in FIG. 2B or may be a memory included in an external device. The processor 210 may select data to be used for retraining from among data generated by a pre-trained machine learning model. As an example, the data used for retraining may include data having a high degree of inconsistency between the result of analysis by the pre-trained machine learning model and a ground truth. Here, the ground truth may include an annotation by the user 24. The high degree of inconsistency between the result of analysis by the pre-trained machine learning model and the ground truth may indicate that the pre-trained machine learning model fails to accurately analyze a medical image. In other words, the high degree of inconsistency between the result of analysis by the pre-trained machine learning model and the ground truth may indicate that the pre-trained machine learning model fails to show the best performance in the second site 20. Accordingly, the processor 210 may retrain the pre-trained machine learning model by using the data having the high degree of inconsistency. The processor 210 may set a threshold value for the degree of inconsistency, and may determine data exceeding the corresponding threshold value as data having a high degree of inconsistency. As another example, the data used for retraining may include data that is highly uncertain about the pre-trained machine learning model. Here, being highly uncertain may indicate that reliability of the result of analysis by the pre-trained machine learning model is low. In the case where the result of analysis by the pre-trained machine learning model has a low probability of clearly corresponding to any one determination (e.g., a determination of whether the result of determination is positive or negative), the case indicates that the reliability of the result of analysis is low. Generating the data having the high uncertainty may indicate that the pre- trained machine learning model fails to show the best performance in the second site 20. Accordingly, the processor 210 may retrain the pre-trained machine learning model by using the data having the high uncertainty. The processor 210 may set a threshold value for uncertainty (or reliability), and may determine data exceeding the corresponding threshold value as data having high uncertainty. As another example, the data used for retraining may include data selected by the user 24 from among the data generated by the pre-trained machine learning model. The user 24 may select the result of analysis of a particular case (e.g., a medical image of a particular patient from among medical images acquired from the second site 20, a medical image corresponding to a particular disease, a medical image of an unusual modality, or the like), the result of analysis of a case in which a pre-trained machine learning model frequently makes errors, the result of analysis of case that is collected a lot from the second site 20, or the like. The processor 210 may retrain the pre-trained machine learning model by using the data selected by the user 24. As another example, the data used for retraining may include data having a low degree of consistency between a first report created by the pre-trained machine learning model and a second report created by the user 24. The low degree of consistency between the first report and the second report may indicate that the pre-trained machine learning model fails to accurately analyze the medical image. Accordingly, the processor 210 may retrain the pre- trained machine learning model by using the data having the low degree of consistency. The processor 210 may set a threshold value for the degree of consistency, and may determine data less than or equal to the corresponding threshold value (data having the degree of consistency less than or equal to the corresponding threshold value ?) as data having a low degree of consistency. As another example, the data used for retraining may include data useful for retraining from among data stored in a medical information system introduced in the second site 20. Here, the data useful for retraining may be data of different types of modalities (e.g., a digital mammography image, an ultrasound image, and the like), but is not limited thereto. In the case where a machine learning model is trained by using data of different types of modalities, accuracy of analysis by the machine learning model may be improved. Accordingly, the processor 210 may retrain the pre-trained machine learning model by using the data of different types of modalities. Referring back to FIG. 3, the processor 210 may retrain the pre-trained machine learning model through any one of various methods below. As an example, the processor 210 may retrain the pre-trained machine learning model on the basis of first labeled data. Here, the first labeled data may include data in which an annotation by a user on a medical image analyzed by the pre-trained machine learning model is reflected. For example, in the case where a medical image including the result of incorrect analysis by the pre-trained machine learning model is output, the user may display a ground truth on the medical image or a report (i.e., may display an annotation). The data in which the annotation by the user is reflected is generated as the first labeled data. In addition, the processor 210 may retrain the pre-trained machine learning model on the basis of the first labeled data. Meanwhile, immediately after the first labeled data is generated, the processor 210 may retrain the pre-trained machine learning model in real time. Alternatively, in the case where the first labeled data is generated, the processor 210 may store the same in the memory 220. In the case where an amount of the first labeled data stored in the memory 220 is accumulated above a certain level, the processor 210 may retrain the pre-trained machine learning model in real time by using the first labeled data stored in the memory 220. As another example, the processor 210 may further include pseudo second labeled data regarding the medical image to retrain the pre-trained machine learning model. In other words, the processor 210 may retrain the pre- trained machine learning model by using the first labeled data and the pseudo second labeled data. Here, the pseudo second labeled data refers to data generated by applying a data-based sampling technique (e.g., a class-aware balancing technique). For example, the first labeled data may indicate a negatively biased feedback (NBF) phenomenon. Accordingly, the processor 210 may generate second labeled data by sampling a non-labeled sample having the same pseudo label as a sample included in the first labeled data. For example, the first labeled data is assumed to be <semantics>{(xlbb,ylbb):b∈Albb,ylbb}<annotation encoding="application / x-tex">\{(x_{lb}^b, y_{lb}^b): b \in A_{lb}^b, y_{lb}^b\}< / annotation>< / semantics> [1..8]} and the second labeled data is assumed to be <semantics>{(xulbb):b∈[1..μ⋅B]}<annotation encoding="application / x-tex">\{(x_{ulb}^b): b \in [1..\mu \cdot B]\}< / annotation>< / semantics>. Here, µ represents a ratio between the second labeled data and the first labeled data. In this case, a supervised learning loss for the first labeled data may be defined as in Equation 1 below. [Equation 1] [Image disponible dans le document PDF, Image available in the PDF document] Here, <semantics>ℋ(x,y)<annotation encoding="application / x-tex">\mathcal{H}(x, y)< / annotation>< / semantics> represents a cross entropy loss, and <semantics>fθ(x,y)<annotation encoding="application / x-tex">f_{\theta}(x, y)< / annotation>< / semantics> represents an output probability of the machine learning model. For the second labeled data, consistency regularization as in Equation 2 below may be achieved by applying weak augmentation <semantics>ω<annotation encoding="application / x-tex">\omega< / annotation>< / semantics> (·) and strong augmentation <semantics>Ω<annotation encoding="application / x-tex">\Omega< / annotation>< / semantics> (·). [Equation 2] [Image disponible dans le document PDF, Image available in the PDF document] Here, <semantics>ŷulbb<annotation encoding="application / x-tex">\hat{y}_{ulb}^b< / annotation>< / semantics> represents a pseudo label obtained from <semantics>fθ(ω(xulb))<annotation encoding="application / x-tex">f_{\theta}(\omega(x_{ulb}))< / annotation>< / semantics>. The processor 210 generates a pseudo-label bank for the second labeled data. For example, at each training iteration during an adaptation process, the processor 210 may select class-aware balancing samples from the bank and consider the same as additional labeled data. The samples may be selected to share the same pseudo label as the first labeled data. Subsequently, the processor 210 may update (i.e., retrain) the machine learning model according to an existing algorithm by using a reconstructed mini batch. Meanwhile, a process of generating the pseudo label bank is as follows. During an adaptation phase, the processor 210 may generate a bank that stores a pseudo label of target data Dt. In detail, the processor 210 may fix the machine learning model before each training epoch and calculate <semantics>Ŷt={(ŷulbn):n∈𝒴t}<annotation encoding="application / x-tex">\hat{Y}_t = \{(\hat{y}_{ulb}^n): n \in \mathcal{Y}_t \}< / annotation>< / semantics> <semantics>[1...Nulb]<annotation encoding="application / x-tex">[1...N_{ulb}]< / annotation>< / semantics>. In addition, the processor 210 may designate a class having the maximum softmax probability as a pseudo label for each sample, as in Equation 3 below. [Equation 3] [Image disponible dans le document PDF, Image available in the PDF document] In addition, the processor 210 may increase reliability of the pseudo label bank by filtering the top p% of samples having a high probability in each class. In addition, a process of selecting a class-aware balancing sample is as follows. The processor 210 may randomly select, from the pseudo label bank, a class-aware balancing sample <semantics>xCaB<annotation encoding="application / x-tex">x_{CaB}< / annotation>< / semantics> having the same pseudo label as labeled data. A sample selected in this way have the same pseudo label as a real label of a sample having a corresponding label (i.e., <semantics>ylb=ŷCaB<annotation encoding="application / x-tex">y_{lb} = \hat{y}_{CaB}< / annotation>< / semantics>). A sample data point having such a pseudo label is added to each mini batch and is considered labeled data. A sampling strategy as described above allows a machine learning model to secure higher robustness from unexpected NBF while maintaining a class distribution within a mini batch. Accordingly, a final loss function may be constructed as in Equation 4 below. [Equation 4] [Image disponible dans le document PDF, Image available in the PDF document] Here, k represents the number of balancing samples. In addition, <semantics>ℒsup<annotation encoding="application / x-tex">\mathcal{L}_{sup}< / annotation>< / semantics> + <semantics>ℒunsup<annotation encoding="application / x-tex">\mathcal{L}_{unsup}< / annotation>< / semantics> represents a loss by a baseline algorithm, and <semantics>1k⋅B∑b=1k⋅Bℋ(ŷCaBb,fθ(xCaBb))<annotation encoding="application / x-tex">\frac{1}{k \cdot B} \sum_{b=1}^{k \cdot B} \mathcal{H}(\hat{y}_{CaB}^b, f_{\theta}(x_{CaB}^b))< / annotation>< / semantics> represents a loss by a class-aware balancing sample. In addition, the processor 210 may retrain the pre-trained machine learning model by using the final loss function. As another example, the processor 210 may retrain the pre-trained machine learning model on the basis of medical information corresponding to the medical image analyzed by the pre-trained machine learning model. Here, the medical information may be acquired by analysis of data stored in a database of a medical information system by the machine learning model. Hereinafter, an example in which the processor 210 retrains a pre-trained machine learning model on the basis of medical information is described with reference to FIG. 6. FIG. 6 is a view illustrating an example in which a processor retrains a pre-trained machine learning model, according to an embodiment. Referring to FIG. 6, the processor 210 may use data stored in a database 620 of a medical information system for retraining. Here, the medical information system refers to a system introduced into a second site 610 (e.g., a hospital). For example, the processor 210 may acquire, from the database 620, the result of analysis (e.g., a medical report on a radiograph image) by the user 24, which corresponds to a medical image. In addition, the processor 210 may extract, through a machine learning model 630, a label on the medical image from the acquired result of analysis. The label on the medical image may be in the form of medical information that may be output when a machine learning model (a different model from the machine learning model 630 of FIG. 6) that learns a plurality of training medical images and medical information corresponding to the plurality of training medical images analyzes the medical image. The medical information, which may be output by the machine learning model, may include whether or not a lesion is present in the medical image, lesion information detected in the medical image, various medical findings other than the lesion, quality and metadata of the medical image (e.g., a modality, capturing information, and the like), quantitative information extracted from the medical image (e.g., the size, volume, ratio, number of lesions, and the like), and the like. The label on the medical image may include a label that is set on the medical image by the user 24. In addition, the processor 210 may use data 640 including the extracted label for retraining for a pre-trained machine learning model. For example, the processor 210 may retrain the pre-trained machine learning model by using the extracted label as a ground truth. Here, the machine learning model 630 may be a model that is different from the pre-trained machine learning model 410 or the retrained machine learning model 420 illustrated in FIGS. 4A, 4B, and 5. For example, the machine learning model 630 may be a large language model (LLM), but is not limited thereto. Referring back to FIG. 3, as another example, the processor 210 may retrain the pre-trained machine learning model on the basis of data satisfying a certain criterion from among the data generated by the pre-trained machine learning model. Here, a computing apparatus that transmits the data generated by the pre-trained machine learning model and a computing apparatus that selects the data satisfying the certain criterion may be distinguished from each other. For example, assuming that an apparatus retraining the pre-trained machine learning model is the first server 11, the second server 21 may transmit, to the first server 11, the data generated by the pre-trained machine learning model. In addition, the first server 11 may select the data satisfying the certain criterion from among the data generated by the pre-trained machine learning model. As another example, the processor 210 may retrain the pre-trained machine learning model on the basis of the data satisfying the certain criterion from among medical data collected from the second site 20. Alternatively, the processor 210 may retrain the pre-trained machine learning model on the basis of data satisfying a certain criterion from among data corresponding to the result of analyzing the medical image collected from the second site 20 through the pre- trained machine learning model. Here, as described above, a computing apparatus that transmits the medical data collected from the second site 20 and a computing apparatus that selects the data satisfying the certain criterion may be distinguished from each other. Hereinafter, an example in which the processor 210 retrains a pre-trained machine learning model by using data satisfying a certain criterion is described with reference to FIG. 7. FIG. 7 is a view illustrating an example in which a processor retrains a pre-trained machine learning model, according to an embodiment. FIG. 7 illustrates a first server 11 as a subject that retrains a pre-trained machine learning model 710. In this case, the first server 11 may acquire the pre- trained machine learning model 710 from the second server 21. Alternatively, the first server 11 may read a pre-trained machine learning model stored in an internal memory. In addition, the first server 11 receives, from the second server 21, data 720 generated by analyzing medical images through the pre-trained machine learning model 710 in the second site 20. The processor 210 may select data satisfying a certain criterion from the data 720. For example, the processor 210 may extract, from the data 720, data needed for retraining for the pre-trained machine learning model 710. Here, the data needed for retraining for the pre-trained machine learning model 710 refers to data through which performance of a retrained machine learning model 730 may be improved over performance of the pre-trained machine learning model 710 by reflecting a data distribution held by the second site 20. For example, as described above with reference to FIG. 5, the data needed for retraining for the pre-trained machine learning model 710 may be data in which the degree of inconsistency between the result of analysis by a pre-trained machine learning model and the result of a diagnosis by a user exceeds a threshold value, data in which reliability of the result of analysis by the pre-trained machine learning model is less than or equal to the threshold value, or medical data regarding patients who match patient information set by the user 24, but is not limited thereto. Meanwhile, the first server 11 may tune at least one hyperparameter or change an architecture by using an automated machine learning (AutoML) technique or the like to improve analysis performance of the pre-trained machine learning model 710. Accordingly, the processor 210 may update the pre-trained machine learning model 710 to generate the retrained machine learning model 730. According to an embodiment of the present disclosure, a machine learning model may be updated by being retrained even without intervention of an expert by using an AutoML technique that automates an existing tuning process performed by an expert of a machine learning model. In the example described above with reference to FIG. 7, the first server 11 is described as acquiring all of the data 720 from the second server 21 and extracting data needed for retraining from the data 720, but is not limited thereto. As an example, the first server 11 may access a memory of the second server 21 to extract and acquire the data needed for retraining from the data 720 stored in the memory. As another example, the first server 11 may provide a certain criterion (i.e., a criterion for extracting data needed for retraining) to the second server 12 in advance. Accordingly, the second server 12 may transmit, to the first server 11, only data satisfying a certain criterion from among the data 720 stored in the memory. Referring back to FIG. 3, the retraining process described above with reference to operation 320 may be performed separately from a process (i.e., an inference process) in which a medical image is analyzed by a pre-trained machine learning model. In other words, the processor 210 may retrain the pre- trained machine learning model separately from the process of analyzing the medical image by the pre-trained machine learning model. Hereinafter, an example in which the processor 210 retrains a pre-trained machine learning model separately from analysis of a medical image is described with reference to FIG. 8. FIG. 8 is a view illustrating an example of an analysis process by a pre- trained machine learning model and a retraining process for the pre-trained machine learning model, according to an embodiment. Referring to FIG. 8, a pre-trained machine learning model 820 may analyze a medical image 810 to output data 830 (e.g., medical information). In other words, the pre-trained machine learning model 820 may be deployed in the second server 21 to perform an inference process. Meanwhile, the retraining process described above with reference to operation 320 may be performed in parallel with the inference process. For example, as the inference process proceeds, the processor 210 may retrain the pre-trained machine learning model 820 by using the data 830. Accordingly, a retrained machine learning model 840 may be generated. As described above, even in the case where the pre-trained machine learning model 820 is retrained, inference by the pre-trained machine learning model 820 is not stopped. Therefore, even while performing retraining for a machine learning model, the second server 21 may analyze a medical image at a time point desired by the user 24 and output data including medical information. Referring back to FIG. 3, in operation 330, the processor 210 evaluates performance of the retrained machine learning model. For example, the processor 210 may generate monitoring information regarding the evaluation of the performance of the retrained machine learning model. In addition, the processor 210 may provide the monitoring information to the user 24. As an example, the processor 210 may evaluate the performance of the retrained machine learning model on the basis of a first index for an area under the curve (AUC) of the retrained machine learning model. The first index may be a value obtained by quantifying an area of a region under a receiver-operating characteristic (ROC) curve representing sensitivity and specificity of the result of analysis by the machine learning model. For example, the processor 210 may generate the result (e.g., a graph) of analyzing a change in the first index for a period of time as monitoring information and provide the same to the user 24. As another example, the processor 210 may evaluate the performance of the retrained machine learning model on the basis of a second index for a distribution of the results of analysis of medical images by the retrained machine learning model. The processor 210 may evaluate the performance of the retrained machine learning model by comparing the distribution of the results of analysis of the medical images by the retrained machine learning model (e.g., a ratio between patients diagnosed as positive and patients diagnosed as negative) with a distribution of the results of analysis previously known for the second site 20. For example, in the case where the second index changes by a certain level or more or less compared to the threshold value, the processor 210 may generate the same as monitoring information and provide the same to the user 24. For example, the second index changing by a certain level or more or changing by a certain level or less may be determined that the performance of the retrained machine learning model is not improved. In general, at the same second site 20, the results of analysis of medical images show a particular distribution. For example, in the case wherein the second site 20 is a higher-level hospital, patients with diseases visit largely, and thus, the results of analysis of medical images generally show that diseases are (i.e., positive) found relatively largely. Meanwhile, in the case where the second site 20 is a health examination center, normal persons visit largely, and thus, the results of analysis of medical images generally show that diseases are not found (i.e., are negative) relatively largely. Therefore, in the case where the second index changes by a certain level or more or changes by a certain level or less from the distribution of the results of analysis previously known for the second site 20, the processor 210 may determine that the performance of the retrained machine learning model is not improved. As another example, the processor 210 may evaluate the performance of the retrained machine learning model on the basis of a third index for the results of analysis of the medical images by the machine learning model before being retrained compared to the results of analysis of the medical images by the retrained machine learning model. For example, the processor 210 may collect the results of analysis by different versions of machine learning models (i.e., a retrained machine learning model and a machine learning models before being retrained) for a period of time, compare the collected results of analysis, and provide the same to the user 24 as monitoring information for a model performance evaluation. Alternatively, the processor 210 may collect the third index for a period of time, generate the result (e.g., a graph) of analyzing a change in the collected third index as monitoring information, and provide the same to the user 24. For example, information (i.e., the third index) obtained by comparing the results of analysis by different versions of machine learning models may include a graph indicating a difference between AUCs of machine learning models, a difference in the frequency of modifications made by the user 24 to the results of analysis output by the machine learning models, the result of analyzing a difference between a report output by the machine learning models and a report finally confirmed by the user 24, and the like. For example, the processor 210 may provide the user 24 with a configuration mode in which a threshold for a difference between the results of analysis by machine learning models may be set. In addition, in the case where the difference between the results of analysis by the machine learning models represents a difference greater than or equal to the threshold, the processor 210 may provide information regarding the same to the user 24. In operation 340, the processor 210 applies the retrained machine learning model on the basis of the result of the evaluation. For example, in the case where an index of the retrained machine learning model satisfies a certain criterion, the processor 210 may apply the retrained machine learning model to the second server 21. Here, the processor 210 may apply the retrained machine learning model to the second server 21 by any one of a first method by which the retrained machine learning model automatically replaces the pre-trained machine learning model and a second method by which the retrained machine learning model replaces the pre-trained machine learning model on the basis of a user input. Hereinafter, an example in which the processor 210 evaluates performance of a retrained machine learning model and applies the retrained machine learning model to the second server 21 on the basis of the result of the evaluation is described with reference to FIG. 9. FIG. 9 is a flowchart illustrating an example in which a processor evaluates performance of a retrained machine learning model and applies the retrained machine learning model to a second server. In operation 910, the processor 210 acquires monitoring information. For example, the processor 210 may generate monitoring information regarding an evaluation of performance of a retrained machine learning model. In addition, the processor 210 may provide the monitoring information to the user 24. Examples in which the processor 210 evaluates the performance of the retrained machine learning model and generates the monitoring information are as described above with reference to operation 330. Therefore, detailed descriptions thereof are omitted below. In operation 920, the processor 210 determines whether or not an index of the retrained machine learning model satisfies a certain criterion. Here, the index satisfying the certain criterion indicates that the performance of the retrained machine learning model is improved beyond a preset threshold. In the case where the index satisfies the certain criterion, the process proceeds to operation 930, and in the case where the index does not satisfy the certain criterion, the process proceeds to operation 940. Here, examples of the index of the retrained machine learning model and whether or not the index satisfies the certain criterion are as described above with reference to operation 330. Therefore, detailed descriptions thereof are omitted below. In operation 930, the processor 210 applies the retrained machine learning model to the second server 21. In the case where the index of the retrained machine learning model is determined in operation 920 as satisfying the certain criterion, the processor 210 applies the retrained machine learning model to the second server 21 by the first method or the second method. For example, the first method may be a method by which the retrained machine learning model automatically replaces the pre- trained machine learning model. In addition, the second method may be a method by which the retrained machine learning model replaces the pre-trained machine learning model on the basis of a user input. For example, the second method may be a method by which the user 24 manually replaces the pre-trained machine learning model with the re-trained machine learning model. Meanwhile, the processor 210 may provide the user 24 with the results of analysis by different versions of machine learning models (i.e., a retrained machine learning model and a pre-trained machine learning model). Here, the user 24 may select any one of retrained machine learning models generated at several time points as a model to be used for analysis of a medical image. In operation 940, the processor 210 does not apply the retrained machine learning model to the second server 21. For example, the processor 210 may drop the retrained machine learning model and perform retraining for the pre- trained machine learning model again. Meanwhile, although not illustrated in FIGS. 1 to 9, the processor 210 may provide the user 24 with various functions related to a machine learning model. As an example, the processor 210 may store retrained machine learning models by version. Accordingly, the user 24 may analyze a medical image by activating a desired version of a machine learning model. As another example, the processor 210 may also provide the user 24 with information regarding comparison of performances of respective versions of retrained machine learning models. FIG. 10 is a diagram illustrating an example of a medical information system that analyzes a medical image. Referring to FIG. 10, a medical information system 1000 may include at least one user terminal 1022, an image storage apparatus 1030, and an image analysis system 1070. The user terminal 1022 has installed therein programs executed by a processor, and includes hardware and software that provide a computing environment and a network environment for performing an operation of the present disclosure. The user terminal 1022 may be, for example, implemented as various types such as a computing device and a mobile device within a workstation. The user terminal 1022 may include a viewer (simply, referred to as a viewer) 1011 that displays medical image-related data stored in the image storage apparatus 1030 by interworking with the image storage apparatus 1030. The viewer 1011 may be, for example, installed and executed on the computing device within the workstation, implemented to access the image storage apparatus 1030, and display the medical image-related data stored in the image storage apparatus 1030. The viewer 1011 is a computer program stored in a computer-readable medium and includes instructions executed by the processor. The processor of the user terminal 1022 may perform an operation described in the present disclosure by executing the instructions. The viewer 1011 may display an image analysis result stored in the image storage apparatus 1030. The viewer 1011 may provide a worklist that is configured in a table format to list and display a list of images that the user needs to read, together with key information. The viewer 1011 may include a picture archiving and communication system (PACS) viewer. Here, the viewer 1011 is a program designed to display the image analysis result stored in the image storage apparatus 1030, and may support an image reading task associated with the worklist, but it is not necessarily limited to a viewer for a reading task. The image storage apparatus 1030 may store and manage captured medical images. Also, the image storage apparatus 1030 may store and manage the results of analysis of the medical images. The image storage apparatus 1030 may include a PACS database. The image storage apparatus 1030 may store data according to a designated data format. For example, the image storage apparatus 1030 may store medical images captured by medical imaging apparatuses according to a digital imaging and communications in medicine (DICOM) standard, and the results of analysis of the medical images, and may communicate with the user terminal 1022 to provide data for image reading. The image storage apparatus 1030 and the viewer 1011 may be configured as a PACS system, and the image storage apparatus 1030 may be a PACS server / DB and the viewer 1011 may be a PACS viewer. In the present disclosure, the DICOM standard used for medical image storage is described as an example, but a medical image standard does not need to be limited to DICOM. The image storage apparatus 1030 may acquire the result of analysis of a medical image from the image analysis system 1070. The result of analysis of the medical image may include various types of medical prediction, including lesion information. The result of analysis of the medical image may be provided to assist the user in image reading, and may be provided as an assistant image in which lesion information is displayed on an image. For example, the assistant image may be a secondary capture (simply, may be referred to as SC) image of DICOM. The SC image is a separate image from an original medical image, which is generated by displaying lesion information on the original medical image, and may be displayed in the PACS viewer. In addition, the result of analysis of the medical image may be provided as a report in a readout document format written in text. For example, the report may be a DICOM basic text structured report (SR). However, the format of the provision of the result of analysis of the medical image is not limited thereto, and may include the results of various types of DICOM formats. Medical images stored in the image storage apparatus 1030 may include images acquired by medical imaging apparatuses of various modalities. The medical image may be an X-ray image, a magnetic resonance imaging (MRI) image, an ultrasound image, a computed tomography (CT) image, a digital mammography (MMG) image, a digital breast tomosynthesis (DBT) image, or the like. In the description, a chest X-ray image is described as an example of a medical image, but the medical image does not need to be limited thereto, and the present disclosure may be applied according to a type of the medical image. The image analysis system 1070 may include at least one of the first server 11 and the second server 21 of FIG. 1. The image analysis system 1070 may analyze a medical image (a target image) requested to be analyzed, by using an artificial intelligence (AI) model and store the result of the analysis in the image storage apparatus 1030. The image analysis system 1070 may include at least one of a machine learning model pre-trained before being deployed in the second site 20 and a machine learning model (i.e., an updated machine learning model) retrained on the basis of data generated by the pre-trained machine learning model, and may output medical information by analyzing an input medical image. The machine learning model is generated to make a medical inference from the input medical image, and a model structure, a training data configuration, a training method, a medical inference object, and the like may be variously designed. As described above with reference to FIGS. 1 to 10, the medical information system 1000 according to the present disclosure may provide a platform capable of enhancing medical image analysis performance by customizing a default machine learning model in a target site (i.e., the second site 20) by an on-premise or cloud method. In addition, the medical information system 1000 may provide a user through the platform with information obtained by continuously monitoring the performance of the machine learning model deployed in the target site and a distribution of medical data held by the target site. In addition, the medical information system 1000 may evaluate, through the platform, the performance of the machine learning model customized in the target site and provide the user with information regarding the result of the evaluation. The platform provided by the medical information system 1000 according to the present disclosure may refer to a basic environment supporting medical image analysis using a machine learning model, and may include hardware and software components that enable a function of the medical information system 1000. In the case where the medical information system 1000 provides the platform driven in the on-premise method, the hardware of the corresponding platform may include a server held by the target site and a user terminal connected to the server through a network. In addition, the software of the corresponding platform may include at least one of a machine learning model that analyzes a medical image and provides the result of the analysis and a viewer that displays medical image-related data. The user terminal may upload medical image data to the server through the viewer, or may receive the result of analysis by the machine learning model from a cloud server and display the same on a screen. Meanwhile, in the case where the medical information system 1000 provides the platform driven in the cloud method, the hardware of the corresponding platform may include a cloud server and a user terminal connected to the cloud server through a network. In addition, the software of the corresponding platform may include at least one of a machine learning model that analyzes a medical image and provides the result of the analysis and a viewer that displays medical image-related data. The user terminal may upload medical image data to the cloud server through the viewer, or may receive the result of analysis by the machine learning model from the cloud server and display the same on a screen. As described above, the medical information system 1000 according to the present disclosure may optimize a retrained machine learning model to output an optimal result in the second site 20 by retraining a default machine learning model by using data in a target site. Accordingly, the user 24 may accurately perform a diagnosis, treatment, and observation of prognosis of a patient by using the output of the retrained machine learning model. Meanwhile, the above-described method may be written as a program that may be executed on a computer and may be implemented in a general- purpose digital computer that operates the program by using a computer- readable recording medium. In addition, a structure of data used in the above- described method may be recorded on a computer-readable recording medium through various means. The computer-readable recording medium may include a storage medium such as a magnetic storage medium (e.g., ROM, RAM, a USB, a floppy disk, a hard disk, or the like) and an optical reading medium (e.g., CD- ROM, DVD, or the like). Those skilled in the art related to the present embodiment may understand that the above-described method may be implemented in a modified form without departing from the essential characteristics of the above description. Therefore, the provided methods should be considered in an illustrative sense rather than a restrictive sense, and the scope of rights should be defined by claims rather than by the foregoing description and should be construed as including all differences falling within the scope equivalent thereto.

Claims

1. A computing apparatus comprising: a memory storing at least one program; and a processor configured to perform at least one operation by executing the at least one program, wherein the processor is further configured to acquire a pre-trained machine learning model generated at a first site, retrain the pre-trained machine learning model on the basis of information related to medical data collected from a second site, evaluate analysis performance of the retrained machine learning model for a medical image, and apply the retrained machine learning model on the basis of a result of the evaluation.

2. The computing apparatus of claim 1, wherein the computing apparatus comprises a first computing apparatus of the first site or a second computing apparatus of the second site.

3. The computing apparatus of claim 1, wherein the computing apparatus is configured to analyze a medical image collected from the second site via the pre-trained machine learning model, and retrain the pre-trained machine learning model on the basis of first labeled data reflecting an annotation by a user on the medical image analyzed by the pre-trained machine learning model.

4. The computing apparatus of claim 3, wherein the computing apparatus retrains the pre-trained machine learning model by further comprising pseudo second labeled data regarding the medical image.

5. The computing apparatus of claim 1, wherein the computing apparatus is further configured to acquire medical information by analyzing a result of a diagnosis of a medical image of a patient by a doctor at the second site by using a large language model, and retrain the pre-trained machine learning model on the basis of the medical image and the medical information.

6. The computing apparatus of claim 1, wherein the computing apparatus is further configured to retrain the pre-trained machine learning model on the basis of data satisfying a certain criterion from among data generated by the pre-trained machine learning model, and the data generated by the pre-trained machine learning model is received from an external apparatus.

7. The computing apparatus of claim 6, wherein the computing apparatus is further configured to automatically tune at least one hyperparameter of the pre-trained machine learning model without intervention of a user.

8. The computing apparatus of claim 1, wherein the computing apparatus is further configured to evaluate medical image analysis performance of the retrained machine learning model together with medical image analysis performance of the pre- trained machine learning model.

9. The computing apparatus of claim 1, wherein the computing apparatus is further configured to evaluate the performance of the retrained machine learning model on the basis of at least one of a first index for an area under the curve (AUC) of the retrained machine learning model, a second index for a distribution of results of analysis known in advance for the second site compared to results of analysis of medical images collected from the second site by the retrained machine learning model, and a third index for results of analysis of the medical images by the pre-trained machine learning model compared to the results of analysis of the medical images by the retrained machine learning model.

10. The computing apparatus of claim 1, wherein the computing apparatus is further configured to apply the retrained machine learning model by any one of a first method by which the retrained machine learning model automatically replaces the pre-trained machine learning model and a second method by which the retrained machine learning model replaces the pre-trained machine learning model on the basis of a user input.

11. A method of customizing a machine learning model, the method comprising: acquiring a pre-trained machine learning model generated at a first site; retraining the pre-trained machine learning model on the basis of information related to medical data collected from a second site; evaluating analysis performance of the retrained machine learning model for a medical image; and applying the retrained machine learning model on the basis of a result of the evaluation.

12. The method of claim 11, wherein the retraining comprises retraining the pre-trained machine learning model by a first computing apparatus of the first site or a second computing apparatus of the second site.

13. The method of claim 11, further comprising analyzing a medical image collected from the second site via the pre-trained machine learning model, wherein the retraining comprises retraining the pre-trained machine learning model on the basis of first labeled data reflecting an annotation by a user on the medical image analyzed by the pre-trained machine learning model.

14. The method of claim 13, wherein the retraining comprises retraining the pre-trained machine learning model by further comprising pseudo second labeled data regarding the medical image.

15. The method of claim 11, further comprising acquiring medical information by analyzing a result of a diagnosis of a medical image of a patient by a doctor at the second site by using a large language model, wherein the retraining comprises retraining the pre-trained machine learning model on the basis of the medical image and the medical information.

16. The method of claim 11, wherein the retraining comprises retraining the pre-trained machine learning model on the basis of data satisfying a certain criterion from among data generated by the pre-trained machine learning model, and a computing apparatus configured to transmit the data generated by the pre- trained machine learning model and a computing apparatus configured to select the data satisfying the certain criterion are distinguished from each other.

17. The method of claim 16, wherein the retraining comprises automatically tuning at least one hyperparameter of the pre- trained machine learning model without intervention of a user.

18. The method of claim 11, wherein the evaluating comprises evaluating medical image analysis performance of the retrained machine learning model together with medical image analysis performance of the pre-trained machine learning model.

19. The method of claim 11, wherein the evaluating comprises evaluating the performance of the retrained machine learning model on the basis of at least one of a first index for an area under the curve (AUC) of the retrained machine learning model, a second index for a distribution of results of analysis known in advance for the second site compared to results of analysis of medical images collected from the second site by the retrained machine learning model, and a third index for results of analysis of the medical images by the pre-trained machine learning model compared to the results of analysis of the medical images by the retrained machine learning model.

20. The method of claim 11, wherein the applying comprises applying the retrained machine learning model by any one of a first method by which the retrained machine learning model automatically replaces the pre-trained machine learning model and a second method by which the retrained machine learning model replaces the pre-trained machine learning model on the basis of a user input.

21. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method of claim 11.