Method and apparatus for customizing machine learning model for medical image analysis

Abstract

A computing device according to one aspect of the present invention comprises: a memory in which at least one program is stored; and a processor for performing at least one operation by executing the at least one program, wherein the processor acquires a pre-trained machine learning model generated at a first site, retrains the pre-trained machine learning model on the basis of information related to medical data collected at a second site, evaluates the performance of the retrained machine learning model, and applies the retrained machine learning model on the basis of the evaluation result.

Description

Method and device for customizing a machine learning model for analyzing medical images

[0001] The present disclosure relates to a method and device for customizing a machine learning model. Specifically, the present disclosure relates to a method and device for customizing a machine learning model for analyzing medical images.

[0002] Recently, technologies are being developed to predict medical information about patients by analyzing medical images using machine learning models. Typically, machine learning models are trained in advance to suit a user's specific needs (e.g., analyzing medical images) and then delivered to the user.

[0003] However, when a pre-trained machine learning model is actually used, performance may vary across users. For example, performance variations can occur for a variety of reasons, including the type of data held by the user and the bias in the data used to train the model.

[0004] The present invention provides a method and device for customizing a machine learning model so that the model can perform optimally according to the user's purpose. Furthermore, the present invention provides a computer-readable recording medium containing a program for executing the above-described method on a computer. The technical challenges to be addressed are not limited to the technical challenges described above, and other technical challenges may exist.

[0005] A computing device according to one aspect includes a memory having at least one program stored therein; and a processor for executing the at least one program to perform at least one operation, wherein the processor obtains a pre-learned machine learning model generated at a first site, re-learns the pre-learned machine learning model based on information related to medical data collected at a second site, evaluates the analysis performance of the re-learned machine learning model for medical images, and applies the re-learned machine learning model based on the evaluation result.

[0006] A method for customizing a machine learning model according to another aspect includes: acquiring a pre-trained machine learning model generated at a first site; re-training the pre-trained machine learning model based on information related to medical data collected at a second site; evaluating the analysis performance of the re-trained machine learning model for medical images; and applying the re-trained machine learning model based on the evaluation result.

[0007] Another aspect of a computer-readable recording medium includes a recording medium having recorded thereon a program for executing the above-described method on a computer.

[0008] FIG. 1 is a diagram illustrating an example of customizing a machine learning model according to one embodiment.

[0009] FIG. 2A is a configuration diagram illustrating an example of a user terminal according to one embodiment.

[0010] FIG. 2b is a configuration diagram illustrating an example of a server according to one embodiment.

[0011] FIG. 3 is a flowchart illustrating an example of a method for customizing a machine learning model according to one embodiment.

[0012] FIGS. 4A and 4B are diagrams illustrating examples of a processor obtaining a pre-trained machine learning model according to one embodiment.

[0013] FIG. 5 is a diagram illustrating examples of data used for retraining a pre-trained machine learning model according to one embodiment.

[0014] FIG. 6 is a diagram illustrating an example of a processor retraining a pre-trained machine learning model according to one embodiment.

[0015] FIG. 7 is a diagram illustrating an example of a processor retraining a pre-trained machine learning model according to one embodiment.

[0016] FIG. 8 is a diagram illustrating an example of an analysis process using a pre-trained machine learning model and a re-training process of a pre-trained machine learning model according to one embodiment.

[0017] FIG. 9 is a flowchart illustrating an example of a processor evaluating the performance of a retrained machine learning model and applying it to a second server according to one embodiment.

[0018] Figure 10 is a drawing illustrating an example of a system for analyzing medical images.

[0019] A computing device according to one aspect includes a memory having at least one program stored therein; and a processor for executing the at least one program to perform at least one operation, wherein the processor obtains a pre-learned machine learning model generated at a first site, re-learns the pre-learned machine learning model based on information related to medical data collected at a second site, evaluates the analysis performance of the re-learned machine learning model for medical images, and applies the re-learned machine learning model based on the evaluation result.

[0020] The terms used in the examples are selected from widely used, current terms, as much as possible. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in the specification should be defined based on their intended meaning and the overall content of the specification, rather than simply their names.

[0021] When a part of the specification is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "unit" and "module" used in the specification mean a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0022] Additionally, terms including ordinal numbers, such as "first" or "second," used in the specification may be used to describe various components, but the components should not be limited by the terms. The terms may be used to distinguish one component from another.

[0023] Hereinafter, "medical image" may refer to an image from which phenotypic features appearing in the human body can be extracted. For example, the medical image may include images of all modalities used in the medical field, such as chest radiograph images, X-ray images, computed tomography (CT) images, positron emission tomography (PET) images, magnetic resonance imaging (MRI) images, ultrasound images, sonography images, functional magnetic resonance imaging (fMRI) images, pathological slide images, mammography (MMG) images, and digital breast tomosynthesis (DBT) images.

[0024] Hereinafter, "pathology slide image" may refer to a whole slide image (WSI) containing a high-resolution image of the entire pathology slide, or may refer to a portion of the whole slide image (e.g., one or more patches or tiles). For example, the pathology slide image may be a digital image scanned through a scanning device (e.g., a digital scanner, etc.).

[0025] A pathology slide image can be divided into a tissue region where tissues collected from a human body are located and a background region, and information about specific proteins, cells, tissues, and / or structures can be obtained from the tissue region.

[0026] Hereinafter, “medical information” may include whether a lesion exists in a medical image, information on lesions detected in the medical image, various medical findings other than lesions, quality and metadata of the medical image (e.g., modality, shooting information, etc.), quantitative information extracted from the medical image (e.g., size, volume, ratio, number of lesions, etc.), etc.

[0027] Additionally, medical information may refer to any medically meaningful information that can be extracted from a medical image. For example, medical information may include at least one of information about a marker in the medical image, an immune phenotype, a genotype, a biomarker score, tumor purity, information about RNA, information about the tumor microenvironment, a treatment method for cancer depicted in the medical image, patient survival information, and treatment outcomes.

[0028] Additionally, medical information may include, but is not limited to, the area, location, size of specific tissues (e.g., cancer tissue, cancer stromal tissue, etc.) and / or specific cells (e.g., tumor cells, lymphocytes, macrophages, endothelial cells, fibroblasts, etc.) within a medical image, diagnostic information of cancer, information related to the likelihood of a subject developing cancer, and / or medical conclusions related to cancer treatment.

[0029] Additionally, medical information may include not only quantifiable values ​​obtained from medical images, but also visualized numerical data, predicted data based on the values, image data, statistical data, and more. For example, medical information may be provided to a user terminal or output through a display device.

[0030] Hereinafter, "biomarker" may refer to an objective and measurable biological indicator that can be used to predict the progression of a disease or the outcome of a treatment. For example, a biomarker may include the expression levels of HER2, TROP2, HER3, DLL3, MET, and FGFR2 proteins, and is not limited to the examples described above, but may also include any biological indicator, such as various proteins expressed from all human genes.

[0031] Below, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be implemented in various different forms and are not limited to the examples described herein.

[0032] FIG. 1 is a diagram illustrating an example of customizing a machine learning model according to one embodiment.

[0033] FIG. 1 illustrates a first server (11), a second server (21), and a user terminal (23). For example, the first server (11) may be placed at a first site (10), and the second server (21) and the user terminal (23) may be placed at a second site.

[0034] For example, the first site (10) may be a company or research institute that analyzes or processes data. The first site (10) may research and develop a machine learning model (12) used to analyze data. The first site (10) may be a supplier that provides the machine learning model (12) to the second site (20).

[0035] The first server (11) located at the first site (10) can train a machine learning model (12). For example, the first server (11) can train the machine learning model (12) using pre-stored data. For example, the data pre-stored at the first server (11) may include, but is not limited to, medical images, medical information, etc.

[0036] The machine learning model (12) learned at the first site (10) can be deployed on the second server (21) so that users (24) of the second site (20) can utilize it. At this time, the second server (21) may be an on-premise server located in the physical space of the second site (20). Alternatively, the second server (21) may be a cloud server that is not located in the physical space of the second site (20), but that the user terminal (23) can access via a network. If the second server (21) is a cloud server, the second server (21) or the user terminal (23) may be connected to the first server (11) via a wireless network.

[0037] For example, a user (24) of a second site (20) can obtain desired information through a machine learning model (22) installed in a second server (21). If the second site (20) is assumed to be a hospital, the second server (21) installed with the machine learning model (22) can analyze a patient's medical image and output various medical information. Accordingly, a user (24) of the second site (20) (e.g., a doctor, pathologist, radiologist, etc.) can perform diagnosis, treatment, observation of the patient's prognosis, etc. through a user terminal (23). At this time, the user terminal (23) and the second server (21) can be connected to each other through a wired or wireless network.

[0038] Here, the machine learning model (22) may be provided from the first site (10). In other words, the machine learning model (12) may be trained through the first server (11), and the trained machine learning model (22) may be loaded onto the second server (21).

[0039] Even if the machine learning model (22) has been trained, the machine learning model (22) may not be able to produce optimal results at the second site (20). For example, assuming that the second site (20) is a hospital or a laboratory, the data stored in the first server (11) may differ from the data collected at the second site (20) due to various factors (e.g., patient information, equipment provided at the hospital, etc.). Therefore, the machine learning model (22) trained with the data stored in the first server (11) may not exhibit optimal performance at the second site (20).

[0040] According to one embodiment, a machine learning model (22) is retrained (or customized) based on data from a second server (21) after being pre-trained using data from a first server (11). That is, the pre-trained machine learning model is retrained using data collected from a second site (20). Accordingly, the re-trained machine learning model can output optimal results from the second site (20). Accordingly, a user (24) can accurately perform patient diagnosis, prognosis observation, medical research, etc. using the output of the re-trained machine learning model.

[0041] Hereinafter, the computing device may be a first server (11), a second server (21), or a user terminal (23).

[0042] For example, the user terminal (23) may be an electronic device including a display device and a device for receiving user input (e.g., a keyboard, a mouse, etc.), and including a memory and a processor. In addition, the display device may be implemented as a touch screen and may perform a function for receiving user input. For example, the user terminal may be, but is not limited to, a notebook PC, a desktop PC, a laptop, a tablet computer, a smart phone, etc.

[0043] For example, the first server (11) or the second server (21) may be a device that communicates with an external device (e.g., a user terminal (23), etc.). For example, the first server (11) or the second server (21) may be a device that stores various data, including medical images, bitmap images corresponding to the medical images, information generated by analysis of the medical images (e.g., information on lesions detected in the medical images, information on findings other than lesions detected in the medical images, medical image interpretation reports, information on at least one object expressed in the medical images, information on at least one biomarker expression, medical information related to the medical images, etc.), quality information of the medical images, information on metadata, and information on a machine learning model used for analysis of the medical images. Alternatively, the first server (11) or the second server (21) may be an electronic device that includes a memory and a processor and has its own computing capability.

[0044] By analyzing medical images using machine learning models, computing devices can identify biological elements (e.g., lesions, medical findings, cancer cells, immune cells, cancer regions, etc.) depicted in the images. These biological elements can be used for histological diagnosis of diseases, prognosis prediction, and treatment decisions.

[0045] Hereinafter, with reference to FIGS. 2a to 10, an example in which a machine learning model is retrained and the retrained machine learning model is applied to a second server (21) or a user terminal (23) will be described.

[0046] As described above, the computing device may be a first server (11), a second server (21), or a user terminal (23). Accordingly, the operations performed by the computing device below may be performed by the first server (11), the second server (21), or the user terminal (23). Alternatively, some of the operations performed by the computing device below may be performed by the first server (11) or the second server (21), and the remainder may be performed by the user terminal (23).

[0047] Below, examples of a user terminal and a server are described with reference to FIGS. 2a and 2b.

[0048] FIG. 2A is a configuration diagram illustrating an example of a user terminal according to one embodiment.

[0049] Referring to FIG. 2A, the user terminal (100) includes a processor (110), a memory (120), an input / output interface (130), and a communication module (140). For convenience of explanation, only components related to the present invention are illustrated in FIG. 2A. Therefore, in addition to the components illustrated in FIG. 2A, other general-purpose components may be further included in the user terminal (100). Furthermore, it will be apparent to those skilled in the art that the processor (110), memory (120), input / output interface (130), and communication module (140) illustrated in FIG. 2A may be implemented as independent devices.

[0050] Additionally, the user terminal (100) may be the user terminal (23) illustrated in FIG. 1.

[0051] The processor (110) can process commands of a computer program by performing basic arithmetic, logic, and input / output operations. Here, the commands may be provided from memory (120) or an external device (e.g., a server (200), etc.). In addition, the processor (110) can generally control the operations of other components included in the user terminal (100). In addition, at least one of the operations of the processor (210), which will be described later with reference to FIG. 2B, may be performed by the processor (110).

[0052] The processor (110) may be implemented as an array of a plurality of logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory storing a program that can be executed on the microprocessor. For example, the processor (110) may include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the processor (110) may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor (110) may refer to a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors coupled with a digital signal processor (DSP) core, or any other such combination of configurations.

[0053] The memory (120) may include any non-transitory computer-readable recording medium. As an example, the memory (120) may include a non-permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, the non-permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be a separate permanent storage device distinct from the memory. In addition, the memory (120) may store an operating system (OS) and at least one program code (e.g., a code for the processor (110) to perform an operation to be described later with reference to FIGS. 3 to 10).

[0054] These software components may be loaded from a computer-readable recording medium separate from the memory (120). This separate computer-readable recording medium may be a recording medium that can be directly connected to the user terminal (100), and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. Alternatively, the software components may be loaded into the memory (120) through a communication module (140) that is not a computer-readable recording medium. For example, at least one program may be loaded into the memory (120) based on a computer program (e.g., a computer program for the processor (110) to perform the operations described below with reference to FIGS. 3 to 10) that is installed by files provided by developers or a file distribution system that distributes installation files of applications through the communication module (140).

[0055] The input / output interface (130) may be a means for interfacing with a device (e.g., a keyboard, a mouse, etc.) for input or output that may be connected to or included in the user terminal (100). In FIG. 2A, the input / output interface (130) is illustrated as an element configured separately from the processor (110), but is not limited thereto, and the input / output interface (130) may be configured to be included in the processor (110).

[0056] The communication module (140) may provide a configuration or function for the server (200) and the user terminal (100) to communicate with each other via a network. In addition, the communication module (140) may provide a configuration or function for the user terminal (100) to communicate with other external devices. For example, control signals, commands, data, etc. provided under the control of the processor (110) may be transmitted to the server (200) and / or external devices via the communication module (140) and the network.

[0057] Meanwhile, although not illustrated in FIG. 2A, the user terminal (100) may further include a display device. Alternatively, the user terminal (100) may be connected to an independent display device via wired or wireless communication to transmit and receive data between the two. For example, medical images, analysis information of medical images, medical information, and additional information based on medical information may be provided to the user (24) via the display device.

[0058] FIG. 2b is a configuration diagram illustrating an example of a server according to one embodiment.

[0059] Referring to FIG. 2B, the server (200) includes a processor (210), a memory (220), and a communication module (230). For convenience of explanation, only components related to the present invention are illustrated in FIG. 2B. Therefore, in addition to the components illustrated in FIG. 2B, other general-purpose components may be further included in the server (200). Furthermore, it will be apparent to those skilled in the art that the processor (210), memory (220), and communication module (230) illustrated in FIG. 2B may be implemented as independent devices.

[0060] Additionally, the server (200) may be the first server (11) or the second server (21) illustrated in FIG. 1.

[0061] The processor (210) can process computer program commands by performing basic arithmetic, logic, and input / output operations. Here, the commands can be provided from memory (220) or an external device (e.g., a user terminal (100), etc.). In addition, the processor (210) can generally control the operations of other components included in the server (200).

[0062] The processor (210) obtains a pre-learned machine learning model. The pre-learned machine learning model may be a machine learning model generated by the first server (11). For example, if the server (200) is the first server (11) of FIG. 1, the processor (210) may read data of the pre-learned machine learning model from the memory (220) or receive data of the pre-learned machine learning model from the second server (21). If the server (200) is the second server (21) of FIG. 1, the processor (210) may read data of the pre-learned machine learning model from the memory (220) or receive data of the pre-learned machine learning model from the first server (11).

[0063] A specific example of the processor (210) acquiring a pre-trained machine learning model is described below with reference to step 310 of FIG. 3.

[0064] For example, the processor (210) may retrain a pre-trained machine learning model based on information related to the second site (20). Here, the information related to the second site (20) may include information related to medical data collected at the second site (20), information generated by or stored in the second server (21), etc. For example, the second server (21) may be equipped with a pre-trained machine learning model. Accordingly, the information related to the second site (20) may also include data generated by the pre-trained machine learning model.

[0065] For example, the processor (210) may retrain a pre-trained machine learning model based on information related to medical data collected at the second site (20). This process may be viewed as a process of customizing the machine learning model to the second site (20). The processor (210) may obtain medical data collected at the second site (20) from an external device or memory (220). The medical data collected at the second site (20) may include at least one of medical data related to a patient's disease, diagnosis, or treatment stored in a database of the second site (20), or medical data related to a patient's disease, diagnosis, or treatment obtained from a patient who visited the second site (20). The medical data collected at the second site (20) may include at least one of medical images, patient information, or medical information that can be extracted from the medical images as information related to a patient's disease, diagnosis, or treatment.

[0066] The processor (210) can customize the machine learning model to the second site (20) by using the medical images collected from the second site (20) and the user's annotations for the medical images to retrain the pre-trained machine learning model. The user's annotations of the second site (20) can be regarded as ground truth optimized for the second site (20). The user's annotations can include whether a lesion exists in the medical image, information on lesions detected in the medical image, various medical findings other than lesions, quality and metadata of the medical image (e.g., modality, shooting information, etc.), quantitative information extracted from the medical image (e.g., size, volume, ratio, number, etc. of lesions), etc.

[0067] Meanwhile, the present disclosure is not limited to an embodiment in which a machine learning model is updated through retraining of a previously trained machine learning model. According to one embodiment, the processor (210) may build a new machine learning model using at least one of medical images collected from the second site (20), user annotations for the medical images, or medical information. According to one embodiment, the processor (210) may evaluate the performance of at least one of the previously trained machine learning model and the retrained machine learning model in analyzing medical data collected from the second site (20), and determine whether to build a new machine learning model based on the evaluation result. If the performance of at least one of the previously trained machine learning model and the retrained machine learning model is below a threshold performance, the processor (210) may determine to build a new machine learning model. Building a new machine learning model may include a process of training the machine learning model using only medical data collected from the second site (20).

[0068] The processor (210) can also utilize medical information other than image format for retraining the machine learning model. The processor (210) can utilize the doctor's diagnosis results from the second site (20) for retraining the machine learning model. For example, the processor (210) can analyze the doctor's diagnosis report on the patient's medical image using a large-scale language model to obtain medical information, and use the obtained medical information for retraining the machine learning model. The processor (210) can retrain a previously trained machine learning model based on the patient's medical image and medical information.

[0069] The processor (210) can retrain a previously learned machine learning model based on data that meets a predetermined standard among the medical data collected at the second site (20).

[0070] According to the various embodiments described above, the processor (210) can customize the pre-trained machine learning model to the second site (20) using various model tuning methods. For example, the processor (210) can perform re-training optimized for the second site by automatically tuning at least one hyperparameter of the pre-trained machine learning model or tuning the model architecture without user intervention using an automated machine learning (AutoML) technique.

[0071] Additionally, the processor (210) may retrain the pre-trained machine learning model based on data generated by the pre-trained machine learning model. For example, the processor (210) may retrain the pre-trained machine learning model using all of the data generated by the pre-trained machine learning model. Alternatively, the processor (210) may select data to be used for retraining from among the data generated by the pre-trained machine learning model. Then, the processor (210) may retrain the pre-trained machine learning model using the selected data.

[0072] As an example, the processor (210) can analyze medical images collected from the second site (20) using a pre-trained machine learning model. The user can customize the machine learning model to the second site (20) by using the analysis results by the pre-trained machine learning model for re-training the pre-trained machine learning model. The processor (210) can re-train the pre-trained machine learning model based on labeled data that reflects the analysis results of the medical images analyzed by the pre-trained machine learning model.

[0073] As another example, the processor (210) may analyze medical images collected from a second site (20) using a pre-trained machine learning model. The user may perform an annotation task to accept, modify, or reject the analysis results by the pre-trained machine learning model. The user's annotations from the second site may be considered ground truth optimized for the second site. The processor (210) may customize the machine learning model to the second site by using the user's annotations to retrain the pre-trained machine learning model. The processor (210) may retrain the pre-trained machine learning model based on first labeled data that reflects the user's annotations for the medical images analyzed by the pre-trained machine learning model.

[0074] The processor (210) can retrain a pre-trained machine learning model based on first labeled data. Here, the first labeled data includes data reflecting a user's annotation of a medical image analyzed by the pre-trained machine learning model.

[0075] As another example, the processor (210) may retrain a pre-trained machine learning model by adding virtual second label data to the first label data (i.e., labeled data or annotated data) that reflects feedback provided by the user. Here, the virtual second label data refers to data sampled by applying a data-based sampling technique (e.g., a class-aware balancing technique). For example, the first label data may exhibit a negatively biased feedback (NBF) phenomenon. Accordingly, the processor (210) may generate the second label data by sampling unlabeled data that has the same virtual labels as the first labeled data (i.e., unlabeled data that has the same pseudo labels as the first labeled data).

[0076] As another example, the processor (210) may retrain a pre-trained machine learning model based on medical information corresponding to a medical image analyzed by the pre-trained machine learning model. Here, the medical information may be obtained by analyzing the machine learning model on data stored in a database of a medical information system (e.g., electronic medical record (EMR), hospital information service (HIS), order communication system (OCS), picture archiving communication system (PACS), etc.). The processor (210) may obtain the medical information by analyzing the data stored in the database of the medical information system using the machine learning model. The processor (210) may retrain the pre-trained machine learning model based on the medical information. At this time, the machine learning model used to obtain the medical information from the data stored in the database of the medical information system may be different from the pre-trained machine learning model and the re-trained machine learning model described in the present disclosure.

[0077] As another example, the processor (210) may retrain a pre-trained machine learning model based on data generated by the pre-trained machine learning model that meets a predetermined criterion. Here, the data generated by the pre-trained machine learning model may be received from an external device. If the server (200) is the second server (21) of FIG. 1, the external device may be the first server (11) or a user terminal (23). If the server (200) is the first server (11) of FIG. 1, the external device may be the second server (21) or a user terminal (23).

[0078] According to the various embodiments described above, the processor (210) can customize a pre-trained machine learning model to the second site (20) using various model tuning methods. For example, the processor (210) can perform re-training optimized for the second site by automatically tuning at least one hyperparameter of the pre-trained machine learning model or tuning the model architecture without user intervention using an automated machine learning (AutoML) technique.

[0079] The processor (210) can retrain the pre-trained machine learning model separately from the process of analyzing (e.g., inferring) the medical image by the pre-trained machine learning model. In other words, the analysis process of the pre-trained machine learning model and the retraining process of the pre-trained machine learning model can be performed independently and in parallel. Therefore, the processor (210) can analyze the medical image using the pre-trained machine learning model and the retrained machine learning model, respectively. The processor (210) can evaluate the medical image analysis performance of the pre-trained machine learning model and the medical image analysis performance of the retrained machine learning model, respectively.

[0080] A specific example of the processor (210) retraining a previously learned machine learning model is described below with reference to step 320 of FIG. 3.

[0081] The processor (210) can evaluate the performance of the retrained machine learning model. For example, the processor (210) can evaluate the performance of the retrained machine learning model based on at least one of a first indicator for the area under the curve (AUC) of the retrained machine learning model, a second indicator for a distribution of analysis results known in advance for the second site (20) compared to the analysis results of medical images of the retrained machine learning model, or a third indicator for the analysis results of medical images of the retrained machine learning model compared to the analysis results of medical images of the previously trained machine learning model.

[0082] A specific example of how the processor (210) evaluates the performance of the retrained machine learning model is described below with reference to step 330 of FIG. 3.

[0083] The processor (210) may apply the retrained machine learning model based on the evaluation results. For example, the processor (210) may apply the retrained machine learning model by either a first method in which the retrained machine learning model automatically replaces the previously trained machine learning model, or a second method in which the retrained machine learning model replaces the previously trained machine learning model based on user input.

[0084] A specific example of applying the retrained machine learning model to the processor (210) is described below with reference to step 340 of FIG. 3.

[0085] The implementation example of the processor (210) is the same as the implementation example of the processor (110) described above with reference to FIG. 2a, so a detailed description is omitted.

[0086] The memory (220) may store various data, such as medical images and data generated according to the operation of the processor (210). In addition, the memory (220) may store an operating system (OS) and at least one program (e.g., a program required for the processor (210) to operate).

[0087] The implementation example of the memory (220) is the same as the implementation example of the memory (120) described above with reference to FIG. 2a, so a detailed description is omitted.

[0088] The communication module (230) may provide a configuration or function for the server (200) and the user terminal (100) to communicate with each other via a network. In addition, the communication module (230) may provide a configuration or function for the server (200) to communicate with another external device. For example, control signals, commands, data, etc. provided under the control of the processor (210) may be transmitted to the user terminal (100) and / or an external device (e.g., another server) via the communication module (230) and the network.

[0089] FIG. 3 is a flowchart illustrating an example of a method for customizing a machine learning model according to one embodiment.

[0090] The method illustrated in FIG. 3 consists of steps that are processed sequentially in the processors (110, 210) illustrated in FIGS. 2A and 2B. Therefore, even if omitted below, the content described above regarding the processors (110, 210) can also be applied to the method illustrated in FIG. 3.

[0091] Additionally, at least one of the steps performed by the processor (210) below may be processed by the processor (110).

[0092] At step 310, the processor (210) obtains a pre-trained machine learning model.

[0093] A pre-trained machine learning model refers to a model whose training has been completed before being loaded onto the second server (21). For example, the pre-trained machine learning model may be a model trained by the first server (11). The first server (11) may train the machine learning model using data stored in memory. The training method is not limited to any one, and the machine learning model may be trained using various methods, including supervised learning or unsupervised learning.

[0094] For example, machine learning models can be used to analyze medical images to generate medical information. Medical images are not limited to a single type and can include images from all modalities used in the medical field.

[0095] A previously trained machine learning model is acquired for retraining. For example, a machine learning model may be trained by a first server (11), and the trained machine learning model may be loaded onto a second server (21) and operated.

[0096] Meanwhile, retraining of a pre-trained machine learning model may be performed on a first server (11) or a second server (21). In other words, the pre-trained machine learning model may be retrained by a first computing device of a first site (10) that generated the pre-trained machine learning model or a second computing device of a second site (20) where the pre-trained machine learning model is deployed. That is, the processor (210) may be a processor of the first server (11) or a processor of the second server (21).

[0097] Below, with reference to Figures 4a and 4b, examples of entities that acquire a pre-trained machine learning model for retraining are described.

[0098] FIGS. 4A and 4B are diagrams illustrating examples of a processor obtaining a pre-trained machine learning model according to one embodiment.

[0099] FIG. 4A illustrates an example in which a second server (21) acquires and retrains a pre-trained machine learning model (410). That is, the second server (21) can acquire a pre-trained machine learning model (410) and generate a re-trained machine learning model (420). In addition, the processor (210) may be included in the second server (21).

[0100] Hereinafter, the creation of a retrained machine learning model (420) means that a previously trained machine learning model (410) is updated through retraining. In other words, the creation of a retrained machine learning model (420) means that a previously trained machine learning model (410) is customized.

[0101] The second server (21) may be a computing device equipped with a pre-trained machine learning model (410). Therefore, the second server (21) does not need to receive the pre-trained machine learning model (410) from an external device. That is, the processor (210) can read the pre-trained machine learning model (410) stored in the memory (220). In addition, the processor (210) can retrain the pre-trained machine learning model (410) to generate a re-trained machine learning model (420).

[0102] FIG. 4b illustrates an example in which the first server (11) acquires a pre-trained machine learning model (410) and retrains the pre-trained machine learning model (410). That is, the first server (11) can acquire a pre-trained machine learning model (410) and generate a re-trained machine learning model (420). In addition, the processor (210) may be included in the first server (11).

[0103] The first server (11) may be a computing device that has trained a machine learning model. Furthermore, the pre-trained machine learning model (410) may be loaded onto the second server (21). As illustrated by the arrow at the bottom of Fig. 4b, the processor (210) of the first server (11) may receive the pre-trained machine learning model (410) from the second server (21). Furthermore, the processor (210) may retrain the pre-trained machine learning model (410) to generate a re-trained machine learning model (420). Furthermore, as indicated by the arrow at the top of Fig. 4b, the processor (210) may transmit the re-trained machine learning model (420) to the second server (21).

[0104] Referring again to FIG. 3, at step 320, the processor (210) retrains the previously trained machine learning model.

[0105] The processor (210) can retrain a pre-trained machine learning model based on information related to the second site (20). For example, the information related to the second site (20) may include information related to medical data collected at the second site (20), information generated by or stored in the second server (21), or data generated by a pre-trained machine learning model.

[0106] As an example, the processor (210) may retrain a pre-trained machine learning model based on information related to medical data collected at the second site (20). For example, the processor (210) may retrain a pre-trained machine learning model based on medical data collected at the second site (20). Alternatively, the processor (210) may retrain a pre-trained machine learning model based on the results of analyzing medical images collected at the second site (20) using the pre-trained machine learning model.

[0107] As another example, the processor (210) can obtain medical information by analyzing a doctor's diagnosis of medical images collected from a second site (20) using a large-scale language model. The processor (210) can retrain a previously trained machine learning model based on the medical images and medical information.

[0108] As another example, the processor (210) may retrain a pre-trained machine learning model based on data that meets certain criteria among medical data collected at a second site. The processor (210) may customize the pre-trained machine learning model to the second site using various model tuning methods. For example, the processor (210) may automatically tune at least one hyperparameter of the pre-trained machine learning model or change the model architecture without user intervention using an automated machine learning (AutoML) technique or the like.

[0109] As another example, the processor (210) may select data to be used for retraining from among data generated by a previously trained machine learning model. Furthermore, the processor (210) may retrain the previously trained machine learning model using the selected data.

[0110] As another example, the processor (210) may retrain a pre-trained machine learning model using all data generated by the pre-trained machine learning model.

[0111] For example, data generated by a pre-trained machine learning model may include various medical information output from medical image analysis. In other words, the data used for retraining may include medical images and the results of analysis of those images (i.e., medical information) by a pre-trained machine learning model.

[0112] Below, with reference to Fig. 5, examples of data used for retraining a previously trained machine learning model are described.

[0113] FIG. 5 is a diagram illustrating examples of data used for retraining a pre-trained machine learning model according to one embodiment.

[0114] Referring to FIG. 5, the processor (210) can retrain a pre-trained machine learning model (410) using data stored in the memory (510). Accordingly, a re-trained machine learning model (420) can be generated. At this time, the data stored in the memory (510) may include various medical information output as medical images are analyzed by the pre-trained machine learning model (410).

[0115] The processor (210) may use at least some of the data stored in the memory (510) for retraining the pre-trained machine learning model (410). For example, the memory (510) may be the memory (220) illustrated in FIG. 2B or may be a memory included in an external device. The processor (210) may select data to be used for retraining from among data generated by the pre-trained machine learning model.

[0116] For example, data used for retraining may include data with a high degree of discrepancy between the analysis results of a previously trained machine learning model and the correct answer (ground truth). Here, the correct answer may include annotations by a user (24).

[0117] A high discrepancy between the analysis results of a pre-trained machine learning model and the correct answer may indicate that the pre-trained machine learning model is unable to accurately analyze medical images. In other words, this may indicate that the pre-trained machine learning model is not performing optimally on the second site (20). Therefore, the processor (210) may retrain the pre-trained machine learning model using data with a high discrepancy.

[0118] The processor (210) can set a threshold for the degree of inconsistency, and can determine that data exceeding the threshold is data with a high degree of inconsistency.

[0119] As another example, data used for retraining may include data with high uncertainty relative to the previously trained machine learning model. High uncertainty here may imply low reliability in the analysis results of the previously trained machine learning model.

[0120] If the analysis results of a pre-trained machine learning model have a low probability of clearly determining a certain judgment (e.g., determining whether the judgment result is positive or negative), this indicates low confidence in the analysis results. The generation of data with high uncertainty may indicate that the pre-trained machine learning model is not performing optimally on the second site (20). Accordingly, the processor (210) may retrain the pre-trained machine learning model using data with high uncertainty.

[0121] The processor (210) can set a threshold for uncertainty (or reliability) and determine data exceeding the threshold as data with high uncertainty.

[0122] As another example, data used for relearning may include data selected by a user (24) from among data generated by a previously learned machine learning model.

[0123] The user (24) can select analysis results for special cases (e.g., medical images of special patients, medical images corresponding to special diseases, medical images of special modalities, etc. among the medical images acquired from the second site (20), analysis results for cases in which the previously learned machine learning model frequently makes errors, analysis results for cases frequently collected at the second site (20), etc.). The processor (210) can retrain the previously learned machine learning model using the data selected by the user (24).

[0124] As another example, data used for retraining may include data with low consistency between a first report created by a pre-trained machine learning model and a second report created by a user (24).

[0125] A low degree of agreement between the first and second reports may indicate that the pre-trained machine learning model is unable to accurately analyze medical images. Therefore, the processor (210) may retrain the pre-trained machine learning model using data with low degrees of agreement.

[0126] The processor (210) can set a threshold for the degree of consistency, and can determine data below the threshold as data with a low degree of consistency.

[0127] As another example, data used for relearning may include data useful for relearning among data stored in a medical information system introduced to the second site (20).

[0128] Here, data useful for retraining may be, but is not limited to, data of different modalities (e.g., digital mammography images and ultrasound images). Training a machine learning model using data of different modalities can improve the accuracy of the machine learning model's analysis. Accordingly, the processor (210) can retrain a previously trained machine learning model using data of different modalities.

[0129] Referring again to FIG. 3, the processor (210) can retrain the pre-trained machine learning model through any of the following various methods.

[0130] As an example, the processor (210) may retrain a pre-trained machine learning model based on first labeled data. Here, the first labeled data may include data reflecting a user's annotation of a medical image analyzed by the pre-trained machine learning model.

[0131] For example, if a medical image containing an incorrectly analyzed result by a pre-trained machine learning model is output, the user can display the correct answer (ground truth) on the medical image or report (i.e., display an annotation). Data reflecting the user's annotation is generated as first label data. Then, the processor (210) can retrain the pre-trained machine learning model based on the first label data.

[0132] Meanwhile, the processor (210) can retrain the pre-trained machine learning model in real time as soon as the first label data is generated. Alternatively, the processor (210) can store the first label data in the memory (220) once it is generated. If the amount of the first label data stored in the memory (220) accumulates to a certain level or more, the processor (210) can retrain the pre-trained machine learning model in real time using the first label data stored in the memory (220).

[0133] As another example, the processor (210) may further include virtual second label data for medical images to retrain the pre-trained machine learning model. In other words, the processor (210) may retrain the pre-trained machine learning model using the first label data and the virtual second label data.

[0134] Here, the virtual second label data refers to data generated by applying a data-based sampling technique (e.g., a class-aware balancing technique). For example, the first label data may exhibit the negatively biased feedback (NBF) phenomenon. Accordingly, the processor (210) may generate the second label data by sampling non-labeled samples having the same virtual labels as the samples included in the first label data.

[0135] For example, the first label data Assuming that the second label data is Here, μ represents the ratio of the second label data to the first label data.

[0136] In this case, the supervised learning loss for the first label data can be defined as in mathematical expression 1 below.

[0137]

[0138] Here, stands for cross entropy loss, refers to the output probability of the machine learning model.

[0139] For the second label data, weak augmentation and strong augmentation By applying , consistency regularization as in mathematical equation 2 below can be achieved.

[0140]

[0141] Here, Is It means a virtual label obtained from .

[0142] The processor (210) generates a pseudo-label bank for the second label data. For example, at each learning iteration during the adaptation process, the processor (210) may select class-aware balancing samples from the bank to be considered as additional label data. These samples may be selected to share the same pseudo-label as the first label data. Thereafter, the processor (210) may update (i.e., retrain) the machine learning model using the reconstructed mini-batches according to an existing algorithm.

[0143] Meanwhile, the process of creating a virtual label bank is as follows. During the adaptation phase, the processor (210) processes target data (D t ) can be created to store virtual labels. Specifically, the processor (210) fixes the machine learning model before each learning epoch, can be calculated. And, the processor (210) can designate the class with the maximum softmax probability for each sample as a virtual label, as in mathematical expression 3 below.

[0144]

[0145] Additionally, the processor (210) can filter out the top p% of samples with high probability in each class to increase the reliability of the virtual label bank.

[0146] In addition, the process of selecting a class-aware balanced sample is as follows. The processor (210) selects a class-aware balanced sample having the same virtual label as the label data in the virtual label bank. can be randomly selected. The samples selected in this way have virtual labels that are identical to the actual labels of the samples with that label (i.e., ) Sample data points with these virtual labels are added to each mini-batch and are considered labeled data.

[0147] This sampling strategy maintains the class distribution within a mini-batch, while ensuring that the machine learning model is more robust against unexpected non-negative biases (NBFs). This allows the final loss function to be constructed as shown in Equation 4 below.

[0148]

[0149] Here, k represents the number of balanced samples. Also, means the loss by the baseline algorithm, means the loss by class-aware balanced samples.

[0150] Additionally, the processor (210) can retrain the pre-trained machine learning model using the final loss function.

[0151] As another example, the processor (210) may retrain a pre-trained machine learning model based on medical information corresponding to a medical image analyzed by the pre-trained machine learning model. Here, the medical information may be obtained by analyzing data stored in a database of a medical information system using the machine learning model.

[0152] Hereinafter, with reference to FIG. 6, an example in which the processor (210) retrains a machine learning model that has been previously learned based on medical information will be described.

[0153] FIG. 6 is a diagram illustrating an example of a processor retraining a pre-trained machine learning model according to one embodiment.

[0154] Referring to FIG. 6, the processor (210) can use data stored in the database (620) of the medical information system for relearning. Here, the medical information system refers to a system introduced to a second site (610) (e.g., a hospital).

[0155] For example, the processor (210) can obtain analysis results (e.g., medical reports for radiological images, etc.) of a user (24) corresponding to a medical image from a database (620). Then, the processor (210) can extract a label for the medical image from the obtained analysis results through a machine learning model (630).

[0156] A label for a medical image may be in the form of medical information that can be output when a machine learning model (different from the machine learning model (630) of FIG. 6) that has learned a plurality of training medical images and medical information corresponding to the plurality of training medical images analyzes the medical image. The medical information that the machine learning model can output may include whether a lesion exists in the medical image, information on lesions detected in the medical image, various medical findings other than lesions, quality and metadata of the medical image (e.g., modality, shooting information, etc.), quantitative information extracted from the medical image (e.g., size, volume, ratio, number of lesions, etc.), etc. The label for the medical image may include a label set by a user (24) for the medical image.

[0157] In addition, the processor (210) can use the data (640) including the extracted labels to retrain a previously trained machine learning model. For example, the processor (210) can retrain a previously trained machine learning model using the extracted labels as ground truth.

[0158] Here, the machine learning model (630) may be a different model from the pre-trained machine learning model (410) or the re-trained machine learning model (420) illustrated in FIGS. 4a, 4b, and 5. For example, the machine learning model (630) may be, but is not limited to, a large language model (LLM).

[0159] Referring again to FIG. 3, as another example, the processor (210) may retrain a previously trained machine learning model based on data generated by the previously trained machine learning model that meets a predetermined criterion. Here, the computing device that transmits the data generated by the previously trained machine learning model and the computing device that selects data that meets the predetermined criterion may be distinguished from each other.

[0160] For example, assuming that the device that retrains a pre-trained machine learning model is the first server (11), the second server (21) can transmit data generated by the pre-trained machine learning model to the first server (11). In addition, the first server (11) can select data that satisfies a predetermined standard from among the data generated by the pre-trained machine learning model.

[0161] As another example, the processor (210) may retrain a pre-trained machine learning model based on data that satisfies a predetermined standard among medical data collected from the second site (20). Alternatively, the processor (210) may retrain a pre-trained machine learning model based on data that satisfies a predetermined standard among data corresponding to the results of analyzing medical images collected from the second site (20) through the pre-trained machine learning model.

[0162] Here, as described above, the computing device that transmits the medical data collected at the second site (20) and the computing device that selects data that meets certain criteria can be distinguished from each other.

[0163] Hereinafter, with reference to FIG. 7, an example in which a processor (210) retrains a previously learned machine learning model using data that satisfies a predetermined standard will be described.

[0164] FIG. 7 is a diagram illustrating an example of a processor retraining a pre-trained machine learning model according to one embodiment.

[0165] In FIG. 7, the first server (11) is depicted as a subject that retrains a pre-trained machine learning model (710). In this case, the first server (11) can obtain the pre-trained machine learning model (710) from the second server (21). Alternatively, the first server (11) can also read the pre-trained machine learning model stored in its internal memory.

[0166] Additionally, the first server (11) receives data (720) generated by analyzing medical images through a machine learning model (710) learned at the second site (20) from the second server (21).

[0167] The processor (210) can select data that satisfies a predetermined standard from the data (720). For example, the processor (210) can extract data necessary for retraining a pre-trained machine learning model (710) from the data (720). Here, the data necessary for retraining the pre-trained machine learning model (710) refers to data that can improve the performance of the re-trained machine learning model (730) over the performance of the pre-trained machine learning model (710) by reflecting the data distribution held by the second site (20). For example, the data necessary for retraining the pre-trained machine learning model (710) may be, but is not limited to, data in which the degree of discrepancy between the analysis result of the pre-trained machine learning model and the user's diagnosis result exceeds a threshold value, data in which the reliability of the analysis result of the pre-trained machine learning model is below a threshold value, or medical data on patients that match the patient information set by the user (24), as described above with reference to FIG. 5.

[0168] Meanwhile, the first server (11) may tune at least one hyperparameter or change the architecture to improve the analysis performance of the pre-trained machine learning model (710) by using an AutoML (automated machine learning) technique, etc. Accordingly, the processor (210) may update the pre-trained machine learning model (710) to generate a re-trained machine learning model (730). According to one embodiment of the present disclosure, by using an AutoML technique that automates the tuning process previously performed by an expert in machine learning models, the machine learning model can be updated through re-training without the intervention of an expert.

[0169] In the example described above with reference to FIG. 7, the first server (11) is described as obtaining all data (720) from the second server (21) and extracting data necessary for relearning from the data (720), but is not limited thereto. As an example, the first server (11) may access the memory of the second server (21) and extract and obtain data necessary for relearning from the data (720) stored in the memory. As another example, the first server (11) may provide the second server (12) with a predetermined standard (i.e., a standard for extracting data necessary for relearning) in advance. Accordingly, the second server (12) may transmit only data that satisfies the predetermined standard from among the data (720) stored in the memory to the first server (11).

[0170] Referring back to FIG. 3, the retraining process described above with reference to step 320 may be performed separately from the process of analyzing medical images using a pre-trained machine learning model (i.e., the inference process). In other words, the processor (210) may retrain the pre-trained machine learning model separately from the process of analyzing medical images using the pre-trained machine learning model.

[0171] Hereinafter, with reference to FIG. 8, an example in which the processor (210) retrains a pre-trained machine learning model separately from analysis of a medical image is described.

[0172] FIG. 8 is a diagram illustrating an example of an analysis process using a pre-trained machine learning model and a re-training process of a pre-trained machine learning model according to one embodiment.

[0173] Referring to FIG. 8, a pre-trained machine learning model (820) can analyze a medical image (810) and output data (830) (e.g., medical information). That is, the pre-trained machine learning model (820) can be installed on a second server (21) and perform an inference process.

[0174] Meanwhile, the retraining process described above with reference to step 320 can be performed in parallel with the inference process. For example, while the inference process is in progress, the processor (210) can retrain the previously trained machine learning model (820) using data (830). Accordingly, a retrained machine learning model (840) can be generated.

[0175] As described above, even when a pre-trained machine learning model (820) is retrained, the inference of the pre-trained machine learning model (820) is not interrupted. Accordingly, even while performing retraining of the machine learning model, the second server (21) can analyze the medical image at a time desired by the user (24) and output data containing medical information.

[0176] Referring again to FIG. 3, at step 330, the processor (210) evaluates the performance of the retrained machine learning model.

[0177] For example, the processor (210) can generate monitoring information for evaluating the performance of a retrained machine learning model. Furthermore, the processor (210) can provide the monitoring information to a user (24).

[0178] As an example, the processor (210) may evaluate the performance of the retrained machine learning model based on a first indicator of the area under the curve (AUC) of the retrained machine learning model. The first indicator may be a quantified value of the area under the ROC (Receiver-Operating Characteristic) curve, which represents the sensitivity and specificity of the analysis results of the machine learning model. For example, the processor (210) may generate a result (e.g., a graph) of analyzing changes in the first indicator over a certain period of time as monitoring information and provide the result to the user (24).

[0179] As another example, the processor (210) may evaluate the performance of the retrained machine learning model based on a second indicator of the distribution of analysis results for medical images of the retrained machine learning model. The processor (210) may compare the distribution of analysis results for the medical images of the retrained machine learning model (e.g., the ratio of patients diagnosed as positive to patients diagnosed as negative) with a previously known distribution of analysis results for the second site (20) to evaluate the performance of the retrained machine learning model. For example, if the second indicator changes by a certain level or more compared to a threshold value, the processor (210) may generate this as monitoring information and provide it to the user (24).

[0180] For example, if the second indicator changes above or below a certain level, it can be judged that the performance of the retrained machine learning model has not improved. Generally, the analysis results of medical images at the same second site (20) exhibit a specific distribution. For example, if the second site (20) is a high-level hospital, there are many visits by patients with existing diseases, so the analysis results of the medical images are relatively often in cases where diseases are found (i.e., positive cases). On the other hand, if the second site (20) is a health checkup center, there are many visits by normal people, so the analysis results of the medical images are relatively often in cases where diseases are not found (i.e., negative cases).

[0181] Accordingly, if the second indicator differs by a certain level or more or by a certain level or less from the previously known analysis result distribution for the second site (20), the processor (210) can determine that the performance of the retrained machine learning model has not improved.

[0182] As another example, the processor (210) may evaluate the performance of the retrained machine learning model based on a third indicator comparing the analysis results of the medical images of the retrained machine learning model with the analysis results of the medical images of the machine learning model before retraining. For example, the processor (210) may collect analysis results of different versions of the machine learning models (i.e., the retrained machine learning model and the machine learning model before retraining) for a certain period of time, compare the collected analysis results, and provide the collected analysis results to the user (24) as monitoring information for evaluating model performance. Alternatively, the processor (210) may collect the third indicator for a certain period of time, generate a result (e.g., a graph) of the analysis of changes in the collected third indicator, and provide the result as monitoring information to the user (24).

[0183] For example, information comparing the analysis results of different versions of machine learning models (i.e., the third indicator) may include a graph showing the difference in AUC of the machine learning models, a difference in the frequency with which the user (24) modified the analysis results output by the machine learning models, and a result of analyzing the difference between the report output by the machine learning models and the report finally confirmed by the user (24).

[0184] For example, the processor (210) may provide the user (24) with a configuration mode that can set a threshold for differences between analysis results of machine learning models. In addition, if the difference between analysis results of machine learning models exceeds the threshold, the processor (210) may provide information about this to the user (24).

[0185] At step 340, the processor (210) applies the retrained machine learning model based on the evaluation results.

[0186] For example, if an indicator of the retrained machine learning model satisfies a predetermined criterion, the processor (210) can apply the retrained machine learning model to the second server (21).

[0187] At this time, the processor (210) can apply the retrained machine learning model to the second server (21) by either the first method in which the retrained machine learning model automatically replaces the pre-trained machine learning model or the second method in which the retrained machine learning model replaces the pre-trained machine learning model based on user input.

[0188] Hereinafter, with reference to FIG. 9, an example in which a processor (210) evaluates the performance of a retrained machine learning model and applies the retrained machine learning model to a second server (21) based on the evaluation result will be described.

[0189] FIG. 9 is a flowchart illustrating an example of a processor evaluating the performance of a retrained machine learning model and applying it to a second server according to one embodiment.

[0190] At step 910, the processor (210) obtains monitoring information.

[0191] For example, the processor (210) can generate monitoring information for evaluating the performance of a retrained machine learning model. Furthermore, the processor (210) can provide the monitoring information to a user (24).

[0192] Examples of how the processor (210) evaluates the performance of a retrained machine learning model and generates monitoring information are as described above with reference to step 330. Therefore, a detailed description thereof is omitted below.

[0193] At step 920, the processor (210) determines whether the indicator of the retrained machine learning model satisfies a predetermined criterion. Here, the indicator satisfying the predetermined criterion means that the performance of the retrained machine learning model has improved beyond a preset threshold.

[0194] If the indicator satisfies the given criteria, proceed to step 930, and if the indicator does not meet the given criteria, proceed to step 940.

[0195] Here, examples of whether the indicators of the retrained machine learning model and whether the indicators satisfy the specified criteria are as described above with reference to step 330. Therefore, a detailed description is omitted below.

[0196] At step 930, the processor (210) applies the retrained machine learning model to the second server (21).

[0197] Through step 920, if it is determined that the indicators of the retrained machine learning model satisfy a predetermined standard, the processor (210) applies the retrained machine learning model to the second server (21) by the first method or the second method. For example, the first method may be a method in which the retrained machine learning model automatically replaces the previously trained machine learning model. In addition, the second method may be a method in which the retrained machine learning model replaces the previously trained machine learning model based on user input. For example, the second method may be a method in which the user (24) manually replaces the previously trained machine learning model with the retrained machine learning model.

[0198] Meanwhile, the processor (210) can provide the user (24) with analysis results of different versions of machine learning models (i.e., a retrained machine learning model and a previously trained machine learning model). At this time, the user (24) can select any one of the retrained machine learning models generated at various points in time as a model to be used for analyzing medical images.

[0199] At step 940, the processor (210) does not apply the retrained machine learning model to the second server (21). For example, the processor (210) may drop the retrained machine learning model and re-perform retraining of the previously trained machine learning model.

[0200] Meanwhile, although not illustrated in FIGS. 1 to 9 , the processor (210) may provide the user (24) with various functions related to the machine learning model. For example, the processor (210) may store retrained machine learning models by version. Accordingly, the user (24) may activate a desired version of the machine learning model to analyze medical images. As another example, the processor (210) may provide the user (24) with information comparing the performance of the retrained machine learning models by version.

[0201] Figure 10 is a diagram illustrating an example of a medical information system that analyzes medical images.

[0202] Referring to FIG. 10, a medical information system (1000) may include at least one user terminal (1022), an image storage device (1030), and an image analysis system (1070).

[0203] The user terminal (1022) is composed of hardware and software that installs programs executed by the processor and provides a computing environment and a network environment for performing the operations of the present disclosure. The user terminal (1022) may be implemented in various types, such as, for example, a computing device within a workstation, a mobile device, etc. The user terminal (1022) may include a viewer (simply referred to as a “viewer”) (1011) that displays medical image-related data stored in the image storage device (1030) in conjunction with the image storage device (1030). The viewer (1011) may be installed and executed, for example, on a computing device within a workstation, and is implemented to connect to the image storage device (1030) and display medical image-related data stored in the image storage device (1030). The viewer (1011) is a computer program stored on a computer-readable medium and includes instructions executed by the processor. The processor of the user terminal (1022) can perform the operations described in the present disclosure by executing commands.

[0204] The viewer (1011) can display the image analysis results stored in the image storage device (1030). The viewer (1011) can be configured in a table format and provide a worklist that lists the images that the user must read along with key information. The viewer (1011) can include a PACS (Picture Archiving and Communication System) viewer. Here, the viewer (1011) is a program created to display the image analysis results stored in the image storage device (1030), and can support the image reading task associated with the worklist, but is not necessarily limited to a viewer for reading tasks.

[0205] The image storage device (1030) can store and manage captured medical images. In addition, the image storage device (1030) can store and manage analysis results for the medical images. The image storage device (1030) can include a PACS database. The image storage device (1030) can store data according to a specified data format. For example, the image storage device (1030) can store medical images captured by medical imaging devices and analysis results of the medical images according to the DICOM (Digital Imaging and Communications in Medicine) standard, and can communicate with a user terminal (1022) to provide data for image interpretation. The image storage device (1030) and the viewer (1011) can be configured as a PACS system, and the image storage device (1030) can be a PACS server / DB and the viewer (1011) can be a PACS viewer.

[0206] In this disclosure, the DICOM standard used for storing medical images is described as an example, but the medical image standard need not be limited to DICOM.

[0207] The image storage device (1030) can obtain analysis results of medical images from the image analysis system (1070). The analysis results of the medical images can include various medical predictions, including lesion information. These analysis results of the medical images can be provided to assist the user in reading the images, and can be provided as auxiliary images that display lesion information on the images. For example, the auxiliary images can be DICOM Secondary Capture (SC) images (simply referred to as SC). The SC images are separate images from the original medical images, created by displaying lesion information on the original medical images, and can be displayed in a PACS viewer. In addition, the medical image analysis results can also be provided as a report in the form of a reading report written in text. For example, the report can be a DICOM Basic Text SR (Structured Report). However, the provision form of the medical image analysis results is not limited thereto, and can include results in various forms of DICOM formats.

[0208] Medical images stored in the image storage device (1030) may include images acquired by medical imaging devices of various modalities. The medical images may be X-ray images, MRI (magnetic resonance imaging) images, ultrasound images, CT (computed tomography) images, Digital MMG (Mammography) images, DBT (digital breast tomosynthesis) images, etc. In the description, a chest X-ray image is described as an example of a medical image, but the medical image need not be limited thereto, and the present disclosure may be applied to any type of medical image.

[0209] The image analysis system (1070) may include at least one of the first server (11) or the second server (21) of FIG. 1. The image analysis system (1070) may analyze a medical image (target image) for which analysis is requested using an artificial intelligence (AI) model, and store the analysis results in the image storage device (1030). The image analysis system (1070) may be equipped with at least one of a pre-trained machine learning model before being installed on the second site (20), or a re-trained machine learning model (i.e., an updated machine learning model) based on data generated by the pre-trained machine learning model, and may output medical information by analyzing an input medical image. The machine learning model is generated to make medical inferences from the input medical image, and the model structure, training data configuration, training method, and medical inference target, etc. may be designed in various ways.

[0210] As described above with reference to FIGS. 1 to 10, the medical information system (1000) according to the present invention can provide a platform that can improve medical image analysis performance by customizing a default machine learning model at a target site (i.e., a second site (20)) in an on-premise or cloud manner. In addition, the medical information system (1000) can continuously monitor the performance of the machine learning model installed at the target site and the distribution of medical data held by the target site through the platform, and provide the user with information. In addition, the medical information system (1000) can evaluate the performance of the customized machine learning model at the target site through the platform and provide the user with information on the evaluation results.

[0211] The platform provided by the medical information system (1000) according to the present invention may mean a basic environment that supports medical image analysis using a machine learning model, and may include hardware and software components that enable the functions of the medical information system (1000).

[0212] When a medical information system (1000) provides a platform operated on-premises, the hardware of the platform may include a server maintained at the target site and a user terminal connected to the server via a network. Furthermore, the software of the platform may include at least one of a machine learning model that analyzes medical images and provides analysis results, or a viewer that displays medical image-related data. The user terminal may upload medical image data to the server via the viewer, or receive analysis results of the machine learning model from a cloud server and display them on the screen.

[0213] Meanwhile, if the medical information system (1000) provides a platform operated in the cloud, the hardware of the platform may include a cloud server and a user terminal connected to the cloud server via a network. Furthermore, the software of the platform may include at least one of a machine learning model that analyzes medical images and provides analysis results, or a viewer that displays medical image-related data. The user terminal may upload medical image data to the cloud server via the viewer, or receive analysis results of the machine learning model from the cloud server and display them on the screen.

[0214] As described above, the medical information system (1000) according to the present invention can optimize the retrained machine learning model to output optimal results at the second site (20) by retraining the default machine learning model using data from the target site. Accordingly, the user (24) can accurately perform diagnosis, treatment, and observation of the prognosis of a patient using the output of the retrained machine learning model.

[0215] Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented on a general-purpose digital computer that runs the program using a computer-readable recording medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.).

[0216] Those skilled in the art will appreciate that the present invention can be implemented in modified forms without departing from the essential characteristics of the above-described invention. Therefore, the disclosed methods should be considered illustrative rather than restrictive. The scope of the claims, not the foregoing description, is defined by the scope of the patent, and should be interpreted to encompass all differences within the scope equivalent thereto.

Claims

1. Memory in which at least one program is stored; and A processor comprising: a processor configured to execute at least one program to perform at least one operation; The above processor, A computing device that acquires a pre-trained machine learning model generated at a first site, retrains the pre-trained machine learning model based on information related to medical data collected at a second site, evaluates the analysis performance of the re-trained machine learning model for medical images, and applies the re-trained machine learning model based on the evaluation results.

2. In paragraph 1, The above computing device, A computing device comprising a first computing device of the first site or a second computing device of the second site.

3. In paragraph 1, The above computing device, A computing device that analyzes medical images collected from the second site using the learned machine learning model, and retrains the learned machine learning model based on first labeled data reflecting a user's annotation of the medical images analyzed by the learned machine learning model.

4. In paragraph 3, The above computing device, A computing device that further includes virtual second labeled data for the medical image to retrain the learned machine learning model.

5. In paragraph 1, The above computing device, A computing device that obtains medical information by analyzing a doctor's diagnosis result for a patient's medical image at the second site using a large-scale language model, and retrains the previously learned machine learning model based on the medical image and the medical information.

6. In paragraph 1, The above computing device, Retraining the above-mentioned machine learning model based on data that satisfies a predetermined standard among the data generated by the above-mentioned machine learning model, Data generated by the above-mentioned machine learning model is received from an external device, a computing device.

7. In paragraph 6, The above computing device, A computing device that automatically tunes at least one hyperparameter of the above-mentioned learned machine learning model without user intervention.

8. In paragraph 1, The above computing device, A computing device for evaluating the medical image analysis performance of the re-learned machine learning model together with the medical image analysis performance of the above-mentioned learned machine learning model.

9. In paragraph 1, The above computing device, A computing device that evaluates the performance of the retrained machine learning model based on at least one of a first indicator for the area under the curve (AUC) of the retrained machine learning model, a second indicator for a distribution of analysis results known in advance for the second site compared to the analysis results of the retrained machine learning model for medical images collected at the second site, or a third indicator for the analysis results of the retrained machine learning model for the medical images compared to the analysis results of the previously trained machine learning model for the medical images.

10. In paragraph 1, The above computing device, A computing device that applies the relearned machine learning model by either a first method in which the relearned machine learning model automatically replaces the previously learned machine learning model or a second method in which the relearned machine learning model replaces the previously learned machine learning model based on user input.

11. Step of obtaining a pre-trained machine learning model generated on the first site; A step of retraining the above-mentioned machine learning model based on information related to medical data collected at a second site; A step of evaluating the analysis performance of the medical image of the above retrained machine learning model; and A method for customizing a machine learning model, comprising: a step of applying the retrained machine learning model based on the evaluation results.

12. In paragraph 11, The above relearning steps are: A method for retraining the previously trained machine learning model by a first computing device of the first site or a second computing device of the second site.

13. In paragraph 11, Further comprising a step of analyzing the medical images collected from the second site using the learned machine learning model; The above relearning steps are: A method for retraining the learned machine learning model based on first labeled data reflecting the user's annotation of the medical image analyzed by the learned machine learning model.

14. In paragraph 13, The above relearning steps are: A method for retraining the above-mentioned machine learning model by further including virtual second labeled data for the above-mentioned medical image.

15. In paragraph 11, Further comprising a step of obtaining medical information by analyzing the doctor's diagnosis results for the patient's medical image at the second site using a large-scale language model; The above relearning steps are: A method for retraining the learned machine learning model based on the medical image and the medical information.

16. In paragraph 11, The above relearning steps are: Retraining the above-mentioned machine learning model based on data that satisfies a predetermined standard among the data generated by the above-mentioned machine learning model, A method wherein a computing device that transmits data generated by the above-mentioned learned machine learning model and a computing device that selects data that satisfies the above-mentioned criteria are distinct from each other.

17. In paragraph 16, The above relearning steps are: A method for automatically tuning at least one hyperparameter of the above-mentioned learned machine learning model without user intervention.

18. In paragraph 11, The above evaluation steps are: A method for evaluating the medical image analysis performance of the re-learned machine learning model together with the medical image analysis performance of the above-mentioned pre-learned machine learning model.

19. In paragraph 11, The above evaluation steps are: A method for evaluating the performance of the retrained machine learning model based on at least one of a first indicator for the area under the curve (AUC) of the retrained machine learning model, a second indicator for a distribution of analysis results known in advance for the second site compared to the analysis results of the retrained machine learning model for medical images collected at the second site, or a third indicator for the analysis results of the retrained machine learning model for the medical images compared to the analysis results of the previously trained machine learning model for the medical images.

20. In paragraph 11, The steps to apply the above are: A method of applying the retrained machine learning model by either a first method in which the retrained machine learning model automatically replaces the previously trained machine learning model or a second method in which the retrained machine learning model replaces the previously trained machine learning model based on user input.

21. A computer-readable recording medium recording a program for executing the method of Article 11 on a computer.

Images